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Designing  effective  decision  support  systems  for  complex  logistics  planning  is  necessarily 
challenging  to  accomplish.  Only  now,  with  the  advent  of  the  Air  Expeditionary  Force  concept  of 
operations  and  impending  advancements  in  logistics  database  capabilities,  is  an  effective 
desktop  decision  support  system  for  production  level  supervisors  a  feasible  reality.  Several 
factors  including  knowledge  about  resources  and  requirements,  uncertainty,  work  domain  task 
constraints,  and  database  capabilities  influence  the  decision  making  process  of  the  logistics 
professional.  To  develop  effective  decision  support  systems,  this  research  proposed  a 
methodology  that  incorporates  image  theoretical  constructs,  work  domain  analysis,  decision 
analysis,  and  knowledge-based  heuristical  modeling  to  capture  and  transform  expert  mental 
frameworks  into  computer-based  support  systems  for  supporting  daily  decision  making. 

Specifically,  this  research  includes  knowledge-based  model  development  and  implementation. 
The  model  development  was  based  on  two  components.  First,  it  was  based  on  field  study 
observations  and  interviews  with  real-world  logisticians  at  two  air  bases.  Second,  it  was  based 
on  decision  making  activity  analysis  using  observation  and  semi-structured  interviews  with 
production  superintendents  of  operational  F-16  maintenance  units  at  Hill  AFB,  UT. 

Two  modes  of  decision  support  were  instantiated  for  the  task  of  aircraft  selection  under  the  Air 
Expeditionary  Force  scenario.  One  mode  simply  presented  information  for  supporting  decision 
making,  while  the  other  provided  more  interactive  decision  support  based  on  the  proposed 
methodology.  These  systems  were  evaluated  using  maintenance  personnel  from  the  445th  Air 
Force  Reserve  Squadron  and  Air  Force  Material  Command  familiar  with  the  elements  of  aircraft 
selection.  Results  showed  that  the  interactive  decision  support  system  significantly  decreased 
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the  time  to  complete  the  task,  but  did  not  conclusively  demonstrate  performance  accuracy 
improvement  or  increased  confidence  in  the  generated  solution.  Overall,  the  results  indicated 
that  the  methodology  developed  produced  a  decision  support  system  that  suggests  tangible 
benefit  for  complex  logistics  planning  in  United  States  Air  Force  activities. 
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1 .0  Introduction 


1.1  Motivation  and  Background 

With  the  emergence  of  ideologically  polarized  world  communities  after  World  War  II,  the  United 
States  Air  Force  (USAF)  has  adhered  to  a  Cold  War  containment  policy  designed  to  win  large- 
scale  conflicts  between  major  world  powers.  These  policies  required  enormous  investments  in 
standing  military  forces,  intelligence,  treaty  maintenance,  and  pre-positioned  supplies.  With  the 
fall  of  the  Soviet  Union,  political  realities  have  caught  up  with  ever  constricting  budgets  of  the 
Department  of  Defense,  resulting  in  the  need  for  a  new  way  of  thinking.  The  challenges  faced  by 
today’s  Air  Force  are  not  only  from  outside  powers  impinging  on  vital  national  interests,  but  also 
from  within  its  own  organizational  structure.  Large-scale  conflicts  require  large  applications  of 
force,  but  the  low  to  mid-level  crises  common  to  our  era  require  proportionate  dispensations  of 
force.  Adhering  to  Cold  War  dogma  increases  the  cost  and  inefficiency  of  modern  forces 
resulting  in  an  “all-or-nothing”  response  to  global  conflicts.  Containment  policies  have 
transformed  into  global  engagement  policies,  which  respond  to  crises  before  they  evolve  into 
larger-scale  conflicts.  This  fundamental  shift  has  resulted  in  corresponding  changes  to  strategic 
implementation  of  force  and  serious  impact  to  logistics  practices  and  methods. 

1.1.1  Fundamental  Shift  in  Air  Force  Strategy:  Expeditionary  Aerospace  Forces 

With  the  advent  of  changing  world  threat  conditions,  increasing  budgetary  constraints,  and  the 
need  for  a  more  responsive,  capable  logistics  capacity,  the  USAF  has  embarked  on  a  new  course 
of  operations  that  mark  the  return  to  expeditionary,  light  and  lean  applications  of  force.  The 
Focused  Logistics  operational  concept  of  Joint  Vision  2010  (JV2010)  and  the  resulting  Global 
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Engagement:  A  Vision  for  the  21st  Century  Air  Force,  outline  the  need  for  a  greater  capacity  to 
launch  world-wide 'military  actions  based  from  the  continental  United  States  (CONUS)  instead  of 
the  present-day  reliance  on  large-scale  forward-basing  of  supplies  and  equipment  overseas.  In 
order  to  meet  the  challenge  of  multiple  low  to  mid-level  crisis  response  efforts,  the  Secretary  and 
Chief  of  Staff  of  the  Air  Force  commissioned  the  USAF  Scientific  Advisory  Board  to  study  and 
make  recommendations  concerning  Air  Expeditionary  Forces  (AEFs).  AEFs  are  defined  to  be 
“tailorable  and  rapidly  employable  air  and  space  assets  that  provide  the  National  Command 
Authority  and  the  theater  commanders-in-chief  with  desired  outcomes  for  a  spectrum  of  missions 
ranging  from  humanitarian  relief  to  joint  or  combined  combat  operations”  (USAF  SAB,  1997,  p.  1). 
Under  the  EAF  concept,  the  Air  Force  is  divided  into  several  Air  Expeditionary  Forces  (AEF), 
each  roughly  equivalent  in  capability,  among  which  deployment  responsibilities  will  be  rotated 
(Tripp  et  al.,  1999).  The  AEF  is  a  unique,  task  organized,  tailorable  warfighting  force  composed 
of  organic  airpower  assets  capable  of  supporting  operations  anywhere  in  the  world  (Goodman, 
1998).  It  provides  the  combatant  commanders  flexible,  rapid  response  force  packages  capable  of 
supporting  a  wide  spectrum  of  operations  while  reducing  the  operations  tempo  (OPTEMPO)  for 
personnel  involved  (Reid,  1999).  This  concept  requires  the  ability  to  deploy  and  employ  quickly, 
adapt  rapidly  to  changes  in  the  scenario,  and  sustain  operations  indefinitely.  To  meet  the 
demanding  timelines,  units  must  be  able  to  deploy  and  set  up  logistics  production  processes 
quickly.  Deploying  units  will,  therefore,  have  to  minimize  deployment  support.  This,  in  turn, 
demands  the  support  system  be  able  to  ensure  the  delivery  of  sufficient  resources  when  needed 
to  sustain  operations  (Reid,  1999). 

1 .1 .2  Using  Computer  Aiding  as  a  Performance-Multiplier 

The  successful  employment  of  crisis  action  plans  (CAP)  for  logistical  operations  in  support  of 
AEFs  depend  heavily  on  the  “transition  from  a  situation  in  which  functional  stovepipes  exchange 
vital  information  late  in  the  cycle,  if  at  all,  to  a  collaborative  environment  supported  by  tools  that 
facilitate  communication  and  decision  making”  (USAF  SAB,  1997,  p.  37).  Part  of  the  solution  is 
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the  incorporation  of  joint  decision  support  tools  that  “will  aggregate,  categorize,  and  depict  data 
elements  in  a  format  easy  to  use  and  understand”  (Joint  Vision  2010,  Focused  Logistics,  1995,  p. 
21).  Decision  support  tools  can  come  in  many  forms  and  may  be  distinguished  by  their  level  of 
interaction  with  the  user.  Low-interaction  support  tools  utilize  the  computer’s  ability  to  perform 
complex  mathematical  processes  in  a  relatively  short  period.  These  low-interaction  support  tools 
depend  heavily  on  the  ability  of  the  developer  to  identify  and  understand  the  relationships 
between  several  objects  of  interest,  expressing  them  in  mathematical  arrangements  that  can  be 
optimized  for  the  best  arrangement  or  combination  of  objects  in  the  system.  Optimized  solution 
generators  attempt  to  take  advantage  of  the  computer’s  ability  to  evaluate  a  significantly  large 
number  of  combinations  to  arrive  at  a  desirable  solution.  Pallet  optimization  routines  that 
generate  the  best  arrangement  of  objects  to  be  shipped  on  a  pallet  exemplify  this  type  of  solution 
generator.  Support  tools  of  this  nature  are  necessarily  non-transparent  to  the  user  and  do  not 
offer  explanations  for  the  decision  rationale.  Optimized  solutions  are  best  used  for  complex 
systems  with  closed-form  analytic  solutions  where  system  objectives  remain  constant.  High- 
interaction  support  tools  rely  upon  the  user’s  ability  to  provide  necessary  processing  of 
information.  A  system  that  determines  and  displays  all  information  relevant  to  the  entire  decision 
process  will  enable  the  user  to  consider  or  disregard  information  at  their  discretion.  Strict  high- 
interaction  support  tools  do  not  attempt  to  confine  the  problem  area  or  offer  solutions,  but  leave 
the  user  to  explore  data  elements  and  determine  a  solution.  Current  logistics  planning  tools,  such 
as  the  Core  Automated  Maintenance  System  (CAMS),  provide  mostly  information  presentation 
and  as  such  do  not  meet  the  need  for  decision  support  tools  outlined  in  Joint  Vision  2010. 
Complexity  in  U.S.  Air  Force  logistics  processes  is  due  in  part  to  the  large  number  and  variety  of 
assets  within  the  system  as  well  as  dynamic  shifts  in  objectives.  Fluid  environments  manifest  in 
warfare  demand  decision-support  tools  that  can  incorporate  changing  objectives  and  complexity. 
The  goal  of  this  study  is  to  develop  and  apply  a  methodology  to  generate  an  interactive  decision 
support  system  for  a  complex  logistics-planning  problem  in  U.  S.  Air  Force  systems. 
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1 .2  Research  Methodology 


Research  methodology  for  this  study  consists  of  four  inter-related  steps.  The  four  steps  are: 
information  gathering,  model  development,  model  application  in  complex  logistical  systems,  and 
model  evaluation  of  performance,  efficiency,  and  trust.  Information  collection  was  accomplished 
by  reviewing  research  literature  in  the  subject  area,  examining  prototypical  maintenance  work 
situations  at  a  local  Air  National  Guard  base,  examining  regulations  and  formal  instructions 
developed  by  the  U.S.  Air  Force,  and  by  conducting  interviews  with  experts  at  Hill  AFB,  Utah. 
User-centered  design  principles  were  used  to  gather  information  in  a  bottom-up  manner. 
Creation  of  the  client  model  involved  two  components,  modeling  air  base  sortie  generation  and 
recovery,  and  capturing  the  decision  process  for  an  instance  of  common  logistical  practice. 
Typical  decision  practices  were  evaluated  for  use  based  on  their  applicability  to  the  logistics 
domain.  The  prototype  model  was  then  validated  using  field-testing  with  subject-matter  experts. 
Subject-matter  experts  were  used  to  gather  subjective  and  objective  data  for  model  evaluation. 
The  primary  focus  of  the  research  was  to  determine  an  approach  that  can  be  used  throughout  the 
logistical  community  to  develop  interactive  computer-based  decision  support  aids  for  AEF-related 
planning  and  execution. 

The  remainder  of  this  thesis  is  as  follows.  Chapter  2  outlines  the  literature  review  of  the  various 
phases  of  this  research.  Chapter  3  describes  the  methodology  applied  to  the  model  development 
of  resource  availability  for  inclusion  in  deployment  activities.  Chapter  4  applies  the  model  to  the 
design  and  evaluation  of  interactive  decision  support  for  logistical  processes.  Finally,  Chapter  5 
discusses  the  contributions  and  limitations  of  this  study  and  outlines  ideas  for  future  research. 
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2.0  Related  Research 


There  are  several  areas  that  are  directly  applicable  to  the  development  of  a  systematic  decision 
support  methodology  in  USAF  logistics.  The  first  section  outlines  a  traditional  generic  framework 
for  developing  a  decision  support  system.  The  framework  serves  to  provide  an  adaptive  iterative 
design  template  from  which  to  implement  decision  support  ideas.  This  framework  is  an  important 
element,  necessary  for  understanding  how  individual  functions  relate  to  the  entire  system 
environment  and  for  identifying  the  user’s  interaction  with  system  components.  The  second  area 
outlines  research  into  database  technologies.  Future  database  solutions  for  USAF  activities, 
programs  underway  and  recent  attempts  to  coordinate  between  a  myriad  of  information  systems 
are  discussed  with  the  emphasis  on  the  state  of  USAF  database  capabilities  and  critical  shortfalls 
for  current  DSS  design.  The  third  section  reviews  system  components  that  make  up  the  model 
base.  The  model  base  outlines  various  techniques  developed  to  manage  and  frame  the  contents 
of  the  database.  These  methods  range  from  strict  system-centered  analytical  tools  that  limit  user 
involvement  to  user-centered  techniques  that  maximize  flexibility  and  allow  for  changing 
environmental  conditions.  The  fourth  and  final  area  discusses  the  perspective  of  the  user  as  a 
fully  integrated  component  within  a  DSS  framework.  Using  Image  theory  to  discern  human 
operator  cognition,  this  section  outlines  important  implications  for  DSS  design.  Image  theory  is  a 
powerful  tool  for  understanding  and  reproducing  mental  representations  in  which  decision-makers 
comprehend  and  interact  with  the  system  environment  (Beach,  1997).  Identification  of  the 
appropriate  user  schema  is  vital  to  the  successful  implementation  of  an  interactive  decision  tool. 
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2.1  Decision  Support  Systems  (DSS) 


Decision  support  systems  change  by  definition  and  function  across  disciplines  in  such  a  way  that 
it  is  necessary  to  first  characterize  the  nature  of  decision  support  systems  before  moving  on  to 
necessary  structural  features.  Steven  Alter  (1980)  examined  fifty-six  systems,  which  supported 
decision-making  varying  by  “degree  of  action  implication  of  system  outputs,  i.e.,  the  degree  to 
which  the  system’s  output  can  directly  determine  the  decision.”  Finding  a  strict  definition  difficult 
to  generate,  he  utilized  this  sample  to  develop  a  set  of  abstractions  describing  their 
characteristics.  Characteristics  of  DSS  differ  from  electronic  data  processing  systems  by  their 
emphasis  on  “increased  individual  and  organizational  effectiveness  rather  than  on  increased 
efficiency  in  processing  masses  of  data”  (Alter,  1980,  p.  3).  Vlatko  Ceric  (1997)  and  Sprague 
(1989)  combined  the  work  of  Alter  (1980)  and  Keen  (1981)  with  their  own  work  to  outline  six 
necessary  qualifications  for  DSSs.  Ceric  summarized  DSS  characteristics  as  being  able  to: 

■  “Assist  the  users  in  semi-structured  decision  tasks, 

■  Support  managerial  judgment, 

■  Improve  the  effectiveness  of  decision  making, 

■  Be  used  by  non-computer  specialists  in  an  interactive  manner, 

■  Combine  use  of  models  with  data  bases,  and 

■  Adapt  to  the  decision-making  approach  of  the  user”  (Ceric,  1 997,  p.  251 ). 

The  above  themes  are  consistent  and  widely  agreed  upon  standards  for  DSS.  They  are  also 
somewhat  ambiguous,  qualifying  most  computer  applications  in  use  today  as  decision  support 
tools  of  one  degree  or  another.  Yet  the  above  list  has  several  aspects  important  to  the  adoption  of 
an  appropriate  logistics  DSS  framework,  namely  semi-structured  task  domains,  the  ability  to 
perform  adaptively  to  the  changing  goals  and  values  of  the  user,  and  support  for  managerial 
judgement. 


6 


Ceric,  Alter,  Keen,  and  Sprague  all  agree  that  semi-structured  decision  tasks  are  the  environment 
in  which  good  DSSs  thrive.  Semi-structured  decision  tasks  are  those  tasks  that  “do  not  display 
enough  structure  to  list  feasible  values  of  parameters....”  (Alter,  1980).  Multiple  criteria  decision 
making  (MCDM)  proponents  further  categorize  semi-structured  decision  spaces  as  having  two 
main  problems,  not  knowing  the  decision  maker  (DM)  value  function  and  the  inability  to  capture 
changing  DM  preferences  as  the  process  of  analyzing  progresses  (Kaliszewski,  1998).  These 
problems  violate,  in  spirit  if  not  in  fact,  the  fourth  and  sixth  characteristic  in  Ceric’s  list.  True 
interaction  and  adaptation  with  the  user  implies  a  level  of  cooperation  between  DSS  components 
not  inherent  in  current  USAF  logistics  systems  (LOCIS,  1997). 

Supporting  managerial  judgment  and  improving  the  effectiveness  of  decision-making  relate  to  the 
decision  space  or  contextual  relationships  within  an  organizational  structure.  Sprague  (1989) 
theorizes  that  because,  improving  performance  is  the  ultimate  objective  and  knowledge  workers 
in  an  organizational  context  are  the  clientele,  any  DSS  framework  must  be  created  adjusting  for 
variation  among  differences  in  organizational  structure.  This  is  not  to  say  that  one  DSS  must 
accommodate  all  levels  of  management  only  that  DSS  must  consider  the  relationship  the  user 
has  with  system  components  within  the  context  of  the  organization.  Herbert  Simon  (1945)  first 
identified  this  concept  as  ‘bounded  rationality’.  Due  to  limited  cognitive  capacity,  the  DM  reduces 
information-processing  demands  by  simplifying  problems  they  encounter.  Therefore,  any  DSS 
framework  must  account  for  the  limiting  capabilities  of  the  user. 

Knowing  the  characteristics  or  qualities  of  DSS  enable  us  to  develop  a  likely  generic  framework  of 
system  components  for  an  appropriate  architecture.  Watson  and  Sprague  (1989)  discuss  useful 
ways  of  thinking  about  component  parts  of  a  DSS  and  the  relationship  among  the  parts. 
Suggested  components  utilize  the  dialog,  data,  and  models  (D,  D,  M)  paradigm.  Component 
architecture  is  shown  in  Figure  1 . 
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Data  Base 


Model 

Base 


Figure  1  shows  the  identity  and  relationships  between  the  three  major  components  of  a  DSS, 
database,  model  base,  and  dialog,  as  they  relate  to  the  DM.  Information  is  made  available 
through  the  database,  interpreted  to  support  particular  decision  spaces  in  the  model  base,  and 

presented  to  the  user  in  the  dialog  or  interface.  Not  shown  in  the  figure  is  the  knowledge  base  or 
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what  the  user  knows  about  the  decision  and  about  how  to  use  the  DSS.  In  USAF  logistics  this 
information  is  largely  treated  as  though  it  comes  from  outside  the  DSS  environment  and  is  not 
traditionally  included  as  a  formal  part  of  the  system. 

2.2  Database  in  Context 

According  to  Captain  R.  Cardona  (personal  communication,  September  9,  1999),  an  F-16 
maintenance  officer  working  for  the  Air  Force  Research  Laboratory,  the  database  in  Figure  1, 
within  the  context  of  USAF  logistics  systems,  is  comprised  of  an  enormous  amount  of  individual 
items.  Examples  of  relevant  data  include  information  concerning  the  status  of  spare  parts, 
personnel,  individual  activities  in  progress,  fuel,  ammunition,  maintenance,  planning  and 
scheduling,  resource  amount  and  availability,  facilities,  and  managerial  policy  and  direction. 
These  data  groupings  interact  in  such  a  way  that  it  is  difficult  to  separate  each  database  one  from 
another.  To  demonstrate  this  and  to  decrease  the  amount  of  confusion  in  explanation,  this 
research  focuses  on  front-line  maintenance  units  that  are  responsible  for  the  sortie  generation  of 
aircraft  in  a  fighter  squadron.  It  is  also  helpful  to  note  that  each  of  these  databases  can 
significantly  differ  based  on  the  type  of  aircraft  or  weapon  system  for  which  an  individual  unit  is 
responsible.  For  example  a  General  Dynamics  F-16  requires  different  parts,  specialists, 
documentation,  fuel,  ammunition,  and  scheduling,  than  does  a  Republic  A-10.  This  information 
can  be  found  by  consulting  the  millions  of  pages  of  documentation  associated  with  the  individual 
aircraft.  This  factor  causes  difficulty  when  each  of  the  individual  weapon  systems  do  not  have 
standardized  information  processing  systems  to  manage  the  data.  The  following  paragraphs  in 
this  section  depict  current  USAF  research  enabling  real-time  update  of  the  database  and 
interaction  among  the  various  database  components. 

In  December  1995,  the  Air  Force  Armstrong  Laboratory  in  concert  with  Paperless  Acquisition 
Initiative  published  a  document  outlining  the  requirements  for  the  Logistics  Command  and  Control 
Information  Support  (LOCIS)  project.  This  project  proposed  to  “assess  the  information 
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requirements,  flow,  and  use  of  current  logistics  management  and  technical  systems”(l_OCIS, 
1995,  p.  1).  The  purpose  of  the  research  was  to  determine  the  “need  for,  and  feasibility  of, 
conducting  a  research  and  development  (R&D)  project  that  would  investigate  better  ways  to 
improve  the  timeliness  and  understandability  of  logistics  information  as  it  feeds  a  wing’s 
command  and  control  process”  (LOCIS,  1995,  p.1).  When  viewing  the  USAF  logistics 
community  informational  data  reservoirs  as  the  database  component  in  Sprague’s  DSS 
framework,  three  of  the  six  major  findings  of  the  study  directly  portray  the  current  state  of  the 
component  for  DSS  use. 

1 .  ‘There  are  numerous  interface  problems  between  logistics  automated  systems.  Most  of 
the  systems  currently  used  do  not  have  basic  clipboard  cut  and  paste  capabilities  that 
users  expect.  [This  bullet  is  seemingly  two  distinct  problems,  (1)  computer  databases  are 
unable  to  communicate  with  each  other  due  to  different  formats,  and  (2)  older  computer 
systems  do  not  meet  with  current  expectations  in  user-interface  design] 

2.  There  is  an  inordinate  amount  of  “fat  finger”  data  collection  occurring  within  all  units.  In 
many  cases  information  is  double  and  triple  entered.  It  is  so  time  consuming  and 
inconvenient  that  data  is  either  not  input  or  is  not  accurate. 

3.  Information  presented  to  decision-makers  is  unintegrated,  not  current,  not  accurate  and 
not  easily  understood.  The  telephone  and  radio  net  are  still  the  most  commonly  used 
systems  for  passing  important  logistics  information  during  normal,  contingency  or 
deployment  operations”  (LOCIS,  1995,  p.5-7). 

These  findings  spawned  efforts  to  revisit  and  renew  research  in  the  area  to  enable  real-time  and 
accurate  information.  One  such  effort  was  the  Integrated  Maintenance  Information  System 
(IMIS).  IMIS  demonstrated  the  “capability  to  access  and  integrate  maintenance  information  from 
multiple  sources  and  present  the  information  to  technicians  through  a  rugged,  hand-held 
computer  (Link,  Von  Holle,  and  Mason,  1987,  p.1).”  IMIS  enabled  workers  at  the  lowest  levels  to 
directly  input  work  order  data  to  a  central  information  database  updated  in  real-time.  IMIS  also 
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incorporated  an  Authoring  and  Presentation  System  (APS).  APS  demonstrated  the  use  of  a 
“neutral”  database,  independent  of  the  computer  system  that  is  used  to  display  the  information. 
Information  entered  by  the  user  is  encoded  independent  of  the  output  format  and  thus  can 
overcome  the  differing  format  issues  associated  with  legacy  presentation  systems.  IMIS  proved 
so  successful  that  an  F-16  system  was  developed  by  Lockheed  to  service  the  F-16C/D  Block 
40/42  aircraft.  The  system,  originally  called  the  F-16  Fault  Reporting/Expert  Diagnostic  (FR/ED) 
System,  is  “designed  to  take  manual  input  or  to  electronically  download  maintenance  data  from 
the  Data  Transfer  Cartridge  (DTC),  installed  in  the  aircraft  by  the  pilot  prior  to  flight  (LOCIS,  1995, 
pg.  37).”  Lockheed,  simultaneous  with  their  development  of  the  F-22  aircraft,  is  also  developing  a 
comparable  fully  integrated  maintenance  information  system. 

Other  current  programs  designed  to  increase  the  availability  and  usefulness  of  real-time  and 
historical  data  are: 

■  Air  Force  Logistics  Information  File  (AFLIF), 

■  Air  Force  Operations  Readiness  Management  System  (AFORMS), 

■  Automated  Weather  Network  (AWN), 

■  Automated  Weather  Distribution  System  (AWDS), 

■  T ransportable  Air  Weather  Distribution  System  (TAWDS), 

■  Base-Level  System  Modernization  (BLSM), 

■  Command  and  Control  Information  Processing  System  (C2IPS),  and  the 

■  Core  Automated  Maintenance  System  (CAMS). 

These  systems  represent  a  determined  attempt  by  the  USAF  to  overcome  current  shortcomings 
in  transparency  and  availability  of  needed  information.  Though  it  is  important  to  note  that  (1)  the 
needed  transparency  and  availability  are  not  far  enough  in  development  to  be  useful  even  in  a 
worker-level  maintenance  DSS,  and  (2)  user  expectations  and  interactions  with  system 
components  are  largely  ignored. 
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2.3  Model  Base  Development 


Models  are  defined  depending  on  their  purpose,  treatment  of  randomness,  and  generality  of 
application.  Banks,  Carson,  and  Nelson  (1996)  define  a  model  to  be  a  representation  of  a  system 
for  the  purpose  of  studying  the  system.  This  definition  can  be  further  refined  for  this  paper,  as  a 
representation  of  a  system  for  the  purpose  of  supporting  decision-making.  This  is  a  very  broad 
definition  and  can  apply  to  a  wide  range  of  techniques  that  make  up  the  model  base.  Sprague 
(1989)  states  that  models  used  for  DSS  can  have  one  of  two  purposes,  optimization  or 
description.  Optimization  models  provide  information  about  points  of  maximization  or 
minimization.  Optimization  is  a  prescriptive  modeling  technique  that  compares  and  evaluates 
decision-making  based  on  some  identifiable  normative  standard.  Beach  (1997)  states  that,  “With 
the  possible  exception  of  structural  modeling,  the  emphasis  in  most  work  on  decision  making  has 
been  on  prescribing  what  should  be  done  rather  than  on  describing  what  decision  makers  actually 
do  and  certainly  not  on  diagnosis  or  implementation”  (p.5).  As  a  result  the  cognitive  process  is 
evaluated  based  on  how  well  it  conformed  to  some  prescriptive,  optimized  models  rather  than  the 
other  way  around.  Descriptive  models  do  not  take  liberties  with  ascribing  a  best  solution,  but  only 
describe  the  system’s  behavior;  not  suggesting  optimizing  conditions.  Considering  Popken’s 
(1992)  assessment,  that  in  logistical  systems  high  levels  of  complexity  and  entity  interactions  as 
well  as  uncertainty  make  strict  analytical  solutions  intractable,  prescriptive  techniques  may  be 
unfounded. 

Models  can  also  vary  in  terms  of  generality  of  application.  Personnel  needing  decision  support 
are  actors  within  the  framework  of  the  organization,  and  as  such  DSS  should  reflect  changes  in 
organizational  hierarchy.  Organizational  officials  at  the  strategic-level  determine  objectives, 
policies,  and  disposition  of  resources.  “Strategic  models  tend  to  be  broad  in  scope  with  many 
variables  expressed  in  compressed,  aggregate  form  (Sprague  and  Watson,  1989,  p.).”  The 
decisions  require  larger  amounts  of  information  from  outside  the  organization  and  project  over  a 
greater  time  period.  In  contrast,  the  lowest  levels,  or  operational  level,  tend  to  require  short-term, 
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internally  generated  information.  To  support  these  activities  researchers  have  generated  several 
techniques  to  assist  the  user  in  devising  better  solutions,  these  include  but  are  not  limited  to 
artificial  intelligence,  genetic  algorithms,  heuristics  and  expert  systems. 

2.3.1  Artificial  Intelligence  (Al) 

Artificial  intelligence  is  the  field  whose  goal  is  to  automate  the  knowledge  process  of  human  users 
through  the  use  of  information  or  knowledge  representations  (Chang,  1985).  Al  exists  in  many 
forms  and  is  often  used  in  the  study  of  mental  faculties  through  the  use  of  computational  models 
(Charniak  and  McDermott,  1984).  Al  is  based  on  the  idea  that  mental  processes  can  be  thought 
of  in  some  level  as  a  kind  of  mathematical,  statistical  or  logical  computation  and  as  such  has 
greatly  increased  understanding  about  how  human  operators  cogitate.  It  is  important  to  note, 
however,  that  even  though  Al  has  its  basis  in  human  cognition,  “it  is  not  committed  to  any 
particular  way  of  producing  the  results  (and  in  particular,  the  methods  may  not  be  exactly  those 
that  people  use)”  (Charniak  and  McDermott,  1984,  p.  7).  Al  systems  use  various  methods  to 
obtain  their  results.  Problem  solving  is  accomplished  using  symbolic  representations  instead  of 
numerical  methods  as  the  basic  unit  of  computation.  These  symbolic  units  are  utilized  in 
algorithmic  frames  so  that  the  incremental  steps  followed  by  the  program  are  influenced  by  the 
particular  problem  being  presented.  Typical  Al  problem-solving  methods  include:  dependency 
directed  backtracking,  problem  decomposition,  generate  and  test,  heuristic  search,  logical 
deduction,  and  meta-reasoning.  Traditional  applications  of  Al  include  search  methods,  robotic 
control,  natural  language  processing,  speech  processing,  strategic  game  playing,  and  pictorial 
information  processing.  Even  though  Al  has  progressed  geometrically  over  the  past  30  years, 
Chang  (1985)  states  that  several  problems,  typically  involving  massive  amounts  of  data,  remain 
to  be  solved  and  advances  in  computing  processing  power  should  facilitate  the  usefulness  and 
power  of  Al  systems.  In  problems  where  massive  data  processing  is  imperative,  these  advances 
have  made  Al  more  accessible  and  practical,  however,  processing  speed  has  not  solved  all 
difficulties,  especially  involving  interactive  human  decision-making  systems. 
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Acquired  influences  on  human  decision-making  can  be  broken  down  into  two  parts,  absolute 
information  and  relative  information  (Kaliszewski,  2000,  p.  162).  Absolute  information  is 
information  about  values  of  separate  criteria.  Relative  information  is  information  about  criteria 
values  relative  changes  when  moving  away  from  a  given  decision  along  a  feasible  direction. 
Relative  information  causes  problems  for  Al  researchers  in  that  “the  decision  maker  (DM)  value 
function  is  usually  not  known,  [and]  DM  preferences  change  as  the  process  of  analysing  a 
decision  problem  progresses”  (Kaliszewski,  2000,  p.  162).  Several  methods  are  available  to 
address  this  problem,  two  of  which  are  Genetic  Algorithms  (GA),  and  Expert  Systems. 

2.3.2  Genetic  Algorithms  (GA) 

Genetic  algorithms  are  a  part  of  evolutionary  computing,  which  is  a  rapidly  growing  area  of 
artificial  intelligence.  GA  originates  in  the  biological  evolutionary  concept.  Each  input  is 
represented  as  a  symbolic  representation  called  a  chromosome.  In  biology,  chromosomes 
combine  during  reproduction  in  such  a  way  that  errors  can  occur  causing  mutation.  In  nature,  the 
degree  to  which  a  combination  survives  can  be  termed  its  “fitness.”  In  GA,  fitness  values  are 
assigned  to  each  new  combination  of  chromosome  pairs  and  tested  to  determine  the  feasibility 
within  the  domain.  Often  the  fitness  value  is  evaluated  using  a  multi-objective  fitness  function 
that  reflects  different  conflicting,  quantifiable  system  goals  and  requirements  such  as  minimizing 
cost  or  maximizing  resource  utilization  (Schneider,  1998;  Narayanan  et  al,  1999).  GAs  are  used 
most  effectively  when  no  “best”  answer  or  condition  is  known  and  when  the  search  for  a  desired 
outcome  is  very  complicated  (Obitko,  1998).  GA  searches  come  from  the  range  of  all  possible 
solutions  and  as  such  can  consume  a  great  deal  of  time  and  are  not  useful  for  time-critical 
solutions  (Obitko,  1998).  AEF  requirements  state  the  need  for  logistics  planning  to  be  completed 
within  24  hours  of  the  notice  for  deployment  and  as  such  can  be  considered  time-critical.  GA 
might  become  more  useful  to  the  logistics  community  with  the  advent  of  more  powerful  computer 
hardware,  but  remains  an  unrealistic  expectation  for  a  current  DSS  application. 
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2.3.3  Expert  Systems 


“Expert  systems,  or  knowledge-based  systems,  are  programs  that  reproduce  the  behavior  of  a 
human  expert  within  a  narrow  domain  of  knowledge”  (Widman  et  al,  1989,  p.  9).  Expert  systems 
usually  consist  of  an  inference  engine  and  a  knowledge  base.  Inference  engines  contain  the 
control  structure  that  enables  the  program  to  use  the  knowledge  base.  Knowledge  base  is  that 
information  captured  from  the  expert  decision-maker  for  use  processing  decisions.  The 
knowledge  base  is  typically  expressed  using  production  rules  or  frames  (Widman  et  al,  1989). 
Production  rules  represent  knowledge  as  a  series  of  “if-then-else”  conditional  operators.  These 
operators  reflect  an  “informal  estimate  of  the  probability  that  the  conclusion  is  true  if  the  premises 
of  the  rule  are  met”  (Widman  et  al,  1989).  Production  rules  add  to  the  modularity,  uniformity,  and 
naturalness  of  the  model,  but  also  detract  from  the  program  efficiency.  Information  framing  deals 
with  the  expert’s  ability  to  extract  a  certain  order  or  mental  model  from  a  situation  (Beach,  1997). 
Representing  an  expert’s  knowledge  framework  is  accomplished  using  values  or  procedures 
stored  in  a  series  that  together  reflect  the  expert  frame.  The  framework  is  not  interactive  and 
relies  heavily  on  captured  knowledge  representations  that  do  not  readily  accommodate  change. 
In  real-life,  frames  are  dynamic.  They  are  continually  generated  and  updated  by  the  expert  in 
such  a  way  that  criteria  have  been  developed  to  help  decide  whether  a  particular  area  of 
knowledge  is  suitable  for  development  of  an  expert  system  (Walters  et  al,  1988): 

■  The  knowledge  required  is  well  defined. 

■  There  exist  people  who  are  acknowledged  experts  in  the  area. 

■  The  experts  can  find  high-quality  solutions  to  a  typical  problem  in  minutes  or  hours,  while 
non-experts  cannot  achieve  equally  good  solution  or  require  much  more  time  to  do  so. 

■  A  timely  solution  of  the  problem  has  high  value. 

■  There  is  little  or  no  requirement  for  commonsense  reasoning. 

■  The  knowledge  base  is  stable. 
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The  knowledge  domain  described  above  is  well  characterized  in  the  military  logistics  arena. 
Countless  regulation  manuals  and  continual  job  training  help  define  the  knowledge  necessary  for 
every  specialty  in  logistics.  Personnel  are  evaluated  using  graduated  training  scales  and 
experience  levels.  For  example,  USAF  military  enlisted  members  are  characterized  as 
apprentices,  journeyman,  craftsman,  or  supervisors.  This  knowledge  hierarchy  enables  the 
researcher  to  readily  identify  ‘supervisors’  as  the  acknowledged  experts  in  each  logistics  career 
field.  These  supervisors  are  available  in  relative  abundance  and  can  be  interviewed  and 
observed  as  needed.  Consistent  with  the  logistics  domain,  supervisors  practice  and  work  under 
the  strictest  time  pressures  in  response  to  military  crises  all  over  the  world.  In  essence,  expert 
knowledge-based  systems  appear  to  be  useful  for  military  logistical  applications,  though  no 
current  USAF  system  could  be  found. 

With  the  identification  of  domain  experts,  it  is  important  to  identify  a  theory  of  decision  making  that 
enables  researchers  to  capture  and  understand  the  user’s  decision-making  schema.  This 
schema  or  frame  is  utilized  to  develop  the  expert  knowledge  base  for  the  model  and 
characterizes  the  necessary  interaction  with  the  user.  Image  theory  provides  a  useful  paradigm 
for  this  purpose. 

2.4  Image  Theory 

2.4.1  Background 

In  order  to  adequately  cover  the  topic  of  Image  theory  it  is  important  to  start  with  the  two 
precursor  theories  from  which  it  derived,  prescriptive  theory  and  behavioral  theory.  Prescriptive 
theory  is  founded  on  directing  decision  making  toward  some  normative  ideal  and  determining 
what  should  be  done  instead  of  what  is  actually  being  done.  Even  though  proponents  of  the 
theory  often  used  language  that  implied  that  prescriptive  theory  paralleled  the  cognitive  process, 
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the  logic  of  prescriptive  models  need  not  conform  to  that  standard  (Beach,  1997).  Behavior  that 
conformed  to  the  models  was  considered  rational,  and  behavior  that  did  not  conform  was  judged 
irrational.  Behavioral  decision  theory  started  as  a  study  of  the  degree  to  which  unaided  human 
decision-making  conforms  to  the  processes  and  output  of  prescriptive  decision  theory.  Decision¬ 
making  was  looked  at  as  a  risk  analysis  or  risk  avoidance  problem.  Theorists  evaluated  through 
experimentation  the  conditions  under  which  a  DM  would  gamble  on  a  difficult  outcome  and  what 
were  the  strategies  involved  in  creating  a  balance  between  payoff  and  loss.  Image  theory 
deviates  from  both  prescriptive  and  behavioral  decision  theory  in  its  almost  exclusive  focus  on 
how  decisions  are  actually  made  (Beach,  1997).  Theorists  of  this  approach  use  observation  to 
diagnose  and  decompose  decision-making  paradigms. 

2.4.2  Constructing  the  DM  Frame 

Beach  outlines  Image  theory  in  the  following  way  (refer  to  Figure  3):  “in  a  nutshell,  decision 
makers  use  their  store  of  knowledge  (images)  to  set  standards  that  guide  decisions  about  what  to 
do  (goals)  and  about  how  to  do  it  (plans)”  (Beach,  1997,  p.  164).  Decision-makers  evaluate  the 
progression  of  a  plan  by  continually  evaluating  it  against  the  goal  to  determine  acceptability. 
Likewise,  any  goals  or  plans  not  conforming  to  the  standards  are  eliminated  from  consideration. 

Images  are  further  defined  in  three  categories,  value  images,  trajectory  images,  and  strategic 
images.  Value  images  refer  to  the  DM’s  internal  principles  and  ethics.  These  images  determine 
the  rules  that  govern  the  DM’s  behavior  within  an  organization  and  with  the  world.  Trajectory 
images  refer  to  the  DM’s  life  goals,  work  goals,  and  in  general  the  direction  of  a  personal  agenda. 
Image  theory  does  not  assume  a  motivated  operator  acting  in  the  best  interest  of  the  organization 
alone,  but  further  differentiates  between  levels  of  motivation.  The  third  image,  strategic, 
anticipates  the  outcome  of  trajectory  images  and  serves  to  resolve  conflicts  between  internal 
images  and  goals.  Cognitive  dissonance,  the  conflict  between  a  person’s  actions  and  beliefs 
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would  be  an  issue  resolved  by  the  strategic  image.  Framing  is  the  process  of  searching  the 
constituent  parts  of  the  three  images  to  find  details  relevant  to  a  particular  problem.  This  frame  is 
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then  used  to  adopt  a  choice  strategy  or  make  decisions.  Choice  strategies  can  be  thought  of  as 
progressing  along  a  continuum  of  cost  or  difficulty.  Strategies  range  from  guessing  (Nonanalytic 
strategies)  to  exhaustive  aided  analytic  strategies.  Unaided  analytic  strategies  or  learned 
procedures  would  fall  in  the  middle  of  the  cost  scale.  Aided  analytic  strategies  might  include 
artificial  intelligence  techniques  that  fully  explore  the  solution  set  while  taking  valuable  time  to 
complete  the  analysis.  The  two  types  of  decisions  in  Image  theory  are  the  adoption  decisions 
and  the  progress  decisions.  Adoption  decisions  “are  about  whether  to  add  new  goals  to  the 
trajectory  image  or  new  plans  to  the  strategic  image"  (Beach,  1997,  p.  168).  Progress  decisions 
determine  whether  a  plan  (or  choice  strategy)  is  progressing  toward  the  fulfillment  of  its  goal. 
Two  types  of  decision  mechanisms,  compatibility  tests  and  profitability  tests  process  these  two 
types  of  decisions.  Compatibility  tests  screen  (similar  to  satisficing)  options  based  on  the  quality 
of  the  decision.  If  the  decision  has  violations  of  standard  above  a  threshold  level  it  is  effectively 
incompatible  with  the  standards  outlined  by  the  choice  strategy  derived  from  the  three  images. 
Compatibility  tests  weight  the  competing  decisions  for  evaluation.  Should  any  decisions  survive 
the  parsing  process  in  the  compatibility  test,  the  profitability  test  determines  which  is  best  based 
on  quantity  of  the  outcomes  associated  with  the  options,  or  the  net  expected  gain.  The  net 
expected  gain  represents  the  difference  between  the  subjective  cost  or  difficulty  and  the  expected 
outcome  utility. 

2.4.3  Image  Theory  Application  to  DSS 

Image  theory  provides  researchers  with  three  important  elements  necessary  for  inclusion  in  a 
logistical  decision  support  system:  (1)  support  for  choice  strategies  identified  as  necessary  to  the 
individual  decision  task,  (2)  support  for  interactive  compatibility  tests,  and  (3)  support  for 
interactive  profitability  tests.  Images  developed  by  the  domain  expert  form  to  create  a  sequential 
iterative  process  for  completing  particular  tasks.  These  tasks  are  to  be  identified  using  a 
descriptive  human-centered  approach  instead  of  using  a  top-down,  normative  prescription  of 
some  ideal  process.  The  identified  process  should  allow  the  user  to  interactively  perform  parsing 
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tasks  (compatibility  tests)  as  well  as  interactive  benefit  analysis  (profitability  tests).  Compatibility 
screening  relies  on  some  absolute  rejection  threshold  to  determine  which  items  under 
consideration  survive.  Profitability  screening,  on  the  other  hand,  is  not  a  single  decision 
mechanism  but  a  collective  term  for  the  individual’s  repertory  of  strategies  for  making  choices  and 
the  mechanism  for  selecting  one  of  those  strategies  for  use  on  a  particular  choice  (Beach,  1997, 
p.  170).  The  rejection  threshold  and  repertory  of  strategies  can  be  identified  by  experimentation, 
observation,  or  interview.  Inclusion  of  these  three  elements  facilitates  the  user’s  natural 
inclination  toward  determining  the  correct  solution  and  as  such  will  simplify  the  decision-making 
process. 

2.5  Summary 

In  this  chapter  several  important  points  are  highlighted.  First,  current  systems  based  on 
traditional  DSS  frameworks  consider  the  user’s  knowledge  and  experience  as  a  component 
outside  of  the  system  environment  with  the  database,  model  base,  and  interface  operating 
without  regard  to  the  user’s  values,  goals,  and  organizational  function.  Second,  there  is  a  need  to 
limit  human-centered  uncertainty  by  using  support  systems  based  on  descriptive  human-centered 
design  principles  rather  than  prescriptive  techniques.  Finally,  knowledge-based  methods  that 
apply  expert  interviews  and  scientific  observation  to  develop  modeling  content  introduce  an 
attractive  alternative  when  combined  with  the  image  theoretic  approach.  This  methodology 
presents  an  appropriate  solution  generator  in  time  critical,  stable  tasks  typified  by  USAF  logistics. 

In  this  thesis,  a  methodology  was  developed  to  determine  an  approach  that  can  be  used 
throughout  the  logistical  community  to  develop  interactive  computer-based  decision  support  aids 
for  AEF-related  planning  and  execution.  The  next  chapter  describes  the  expert  model 
development  and  application  of  the  methodology  to  an  AEF-related  task. 
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3.0  Research  Methodology 


3.1  Introduction 

This  thesis  developed  a  human-centered  image  theoretic  methodology  as  an  iterative  design 
approach  to  the  development  of  a  logistics  DSS.  The  research  methodology  consisted  of  two 
major  phases:  knowledge-based  model  development  and  implementation.  Model  development 
examined  prototypical  situations  to  extract  a  suitable  mental  schema  allowing  sufficiently  general 
application  to  individual  problems.  The  mental  schema  for  the  model  development  was  based  on 
two  major  components:  (1)  work  domain  analysis  using  open-ended  interviews  and  observations 
at  the  Springfield,  OH  Air  National  Guard  Base  and  at  the  ACC  Logistics  Readiness  Training 
Center  at  Hill  AFB,  UT,  and  (2)  decision  making  activity  analysis  using  observation  and  semi- 
structured  interviews  with  Production  Superintendents  of  operational  F-16  maintenance  units  at 
Hill  AFB,  UT.  Implementation  of  the  model  was  accomplished  using  Sun  Microsystems’s  object- 
oriented  programming  language  Java  to  develop  a  PC-based  desktop  DSS. 

3.2  Model  Development 

Model  development  utilized  work  domain  analysis  techniques  to  elicit  an  appropriate 
representation  of  a  work  domain  that  is  useful  for  system  design  and  analysis.  The  objective  is 
not  to  represent  the  actual  interaction  among  the  various  system  components  for  a  specific 
situation,  but  to  produce  a  generalized  representation  of  the  work  domain  in  terms  of  functions, 
activities,  and  trajectory  (Rasmussen,  1994).  These  elements  comprise  the  environment  in  which 
the  human  operator  conducts  business  within  the  organization.  By  identifying  these  elements,  a 
task-specific  DSS  can  be  implemented  that  is  independent  of  the  various  image  states,  types  of 


21 


decisions,  and  tests  conducted  by  the  human  operator  as  outlined  by  Image  Theory.  Analysis  is 
accomplished  by  identifying  three  levels  of  abstraction,  general  work  activities  and  functions, 
problem  specific  processes  and  activities,  and  individual  task  decision-making  activities. 

3.2.1  General  Work  Activities  and  Functions 

The  decision-makers  relative  placement  and  function  within  the  organization,  relationship  to  other 
intra-organizational  entities,  and  direction  from  higher  echelons  heavily  influence  the  development 
of  a  knowledge-based  model  of  decision  making.  This  section  outlines  the  work  activities  and 
functions  of  an  Operations  Squadron  maintenance  unit  that  are  typical  of  the  Objective  Wing 
Structure  in  USAF  Air  Combat  Command  (ACC).  The  denotation  of  activity  or  function  at  this 
level  of  abstraction  is  independent  of  the  underlying  processes  involved  as  well  as  their  physical 
implementation. 

The  Objective  Wing  concept  is  a  decentralized  structure  designed  to  maximize  mission 
effectiveness  by  anticipating  a  combat  environment  characterized  by  deployed  USAF  flying  units 
producing  sustained  aircraft  sortie  rates  under  hostile  fire  conditions  (ACCI  21-101).  In  order  to 
achieve  decentralization,  the  organization  further  subdivides  maintenance  organizations  into  on- 
equipment  sortie  production  (Operations)  and  off-equipment  support  (Logistics).  These 
designations  do  not  delineate  between  logistical  and  non-logistical  operations,  but  rather  identify 
broad  categories  of  functional  concentrations.  For  example,  maintenance  units  exist  within  the 
Operations  Group  and  also  in  the  Logistics  Group.  Maintenance  units  that  repair  and  maintain 
individual  aircraft  systems  are  located  in  the  Logistics  Group  and  generally  concentrate  on  those 
activities  that  require  higher  expertise  with  individual  system  components.  Operations  Group 
maintenance  units  are  responsible  for  the  day-to-day  planning,  scheduling,  repairing,  inspecting, 
and  documenting  of  aircraft  in  coordination  with  the  flying  component  resident  in  the  Operations 
Squadron  (OS).  Repair  actions  beyond  the  capabilities  of  the  Operations  Maintenance  unit  are 
handled  by  the  Logistics  Group.  The  overarching  goal  of  both  groups  is  the  same,  however, 
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enabling  safe,  consistent  and  reliable  operation  of  aircraft  for  the  purpose  of  accomplishing  a 
specified  mission.  The  Operations  Squadron  organizational  chart  is  depicted  in  Figure  4  and 
should  be  used  as  each  maintenance-related  activity  and  function  is  explained.  Relevant 
structural  activities  and  functions  include  the  Squadron  Maintenance  Officer,  Squadron  PS&D, 
the  Maintenance  Operations  Center  (MOC),  and  elements  of  the  Sortie  Generation  Flight. 


Figure  3:  Organizational  Chart  of  Objective  Wing  Operations  Squadron  (Adapted 
from  ACCI  21-101,  p.  31,  public  domain) 
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The  Squadron  Maintenance  Officer  (SMO)  is  responsible  to  the  Operations  Squadron 
Commander  for  maintenance  production  as  outlined  in  ACCI  21-101,  Section  2.6.  The  SMO, 
assisted  by  the  Maintenance  Superintendent  (MS),  manages  resources  necessary  to  accomplish 
the  mission.  The  SMO  and  the  Operations  Officer  serve  as  the  liaison  between  the  needs  of  the 
OS  flying  components  and  the  OS  maintenance  components.  The  needs  of  the  flying 
components  include  successfully  achieving  flying  hour  goals  and  maintaining  current  pilot 
qualification  in  the  mission  design  series.  The  needs  of  the  maintenance  component  include 
time,  personnel,  resources,  and  facilities  to  inspect,  document,  and  repair  aircraft  for  routine 
operations.  Coordination  is  a  key  element  to  the  smooth  function  of  flying  operations  and  as  such 
the  SMO  and  MS  spend  a  great  deal  of  time  meeting  with  cross-functional  areas  to  resolve 
personnel,  supply  and  production  issues.  The  SMO  and  MS  perform  the  following  functions  as 
outlined  by  ACCI  21-101: 

■  Designate  Maintenance  Flight  Commanders/  Chiefs. 

■  Implement  Monthly  Maintenance  Plans. 

■  Monitor  Dedicated  Crew  Chief  (DCC)  Program  and  certifies  the  Aircraft  Dedicated 
Crew  Chief  in  writing. 

■  Ensure  personnel  are  qualified  to  support  OS  tasking. 

■  Establish  procedures  for  the  Structural  Integrity  Program. 

■  Periodically  review  CAMS  data. 

■  Publish  procedures  covering  storage,  control  and  handling  of  starter  cartridges  to 
meet  daily  alert,  training,  and  single  integrated  operations  plan  (SIOP)  requirements. 

■  Ensure  assigned  personnel  understand  the  purpose  of  AF  Form  2409,  General 
Sequence  Action  Schedule. 

■  Monitor  oil  analysis  program  (OAP)  status. 
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The  Squadron  Maintenance  Plans,  Scheduling  and  Documentation  (OS  PS&D)  Section  reports 
directly  to  the  SMO/  Superintendent,  and  is  the  focal  point  for  all  squadron  maintenance  planning. 
OS  PS&D  performs  all  scheduling  duties  for  assigned  aircraft,  and  maintains  a  liaison  between 
the  squadron,  operations  support  squadron  PS&D,  and  the  Logistics  Group  Mission  Analysis 
Section  (LSS/EM).  The  planning  process,  at  the  squadron  level,  is  a  consolidated  task  involving 
all  squadron  supervisors.  Automated  data  collection  products,  like  CAMS,  are  used  to  forecast, 
schedule,  and  monitor  completion  of  squadron  aircraft  hourly  inspections,  special  inspections, 
technical  orders,  and  replacement  of  time  change  items.  OS  PS&D  responsibilities  as  outlined  by 
ACCI  21-101  include: 

■  Plan  and  schedule  the  use  of  squadron  aircraft  to  meet  flying  requirements. 

■  Conduct  unit  pre-dock  and  attend  daily  maintenance  meetings. 

■  Perform  the  PS&D  portion  of  aircraft  document  reviews. 

■  Initiate  and  maintains  folders  for  applicable  TCTOs. 

■  Ensure  major  maintenance  support  requirements  are  loaded  into  CAMS. 

■  Maintain  aircraft  historical  documents. 

■  Compute  OS  Maintenance  Planning  Effectiveness  and  forwards  data  to  analysis 
section. 

■  Attend  various  meetings  that  ensure  proper  coordination. 

The  Maintenance  Operations  Center  (MOC)  monitors  sortie  production,  maintenance  production, 
and  execution  of  the  flying  and  maintenance  schedules  provided  by  the  PS&D  Section.  The 
MOC  sets  priorities  for  their  respective  production  efforts  to  meet  mission  requirements.  Priorities 
are  set  for  activities  such  as  fuel  or  calibration  docks,  wash  racks,  and  dispatched  specialists  from 
the  maintenance  squadron.  The  exchange  of  information  between  squadrons  and  the  MOC  must 
be  in  sufficient  detail  to  allow  the  MOC  to  comply  with  reporting  requirements  and  to  identify 
potential  problems.  MOC  responsibilities  as  outlined  in  ACCI  21-101  include: 


25 


*  Maintain  visual  aids  that  show  the  status  and  location  of  each  aircraft  on  station, 
maintained,  or  supported  by  the  Wing. 

■  Ensure  aircraft  status  is  properly  reported  and  maintained. 

■  Coordinate  and  monitor  the  progress  of  Aircraft  Functional  Check  Flights. 

■  Inform  affected  activities  of  changes  in  priorities,  plans,  and  schedules. 

■  Coordinate  changes  to  the  flying  schedule. 

■  Coordinate  munitions  delivery  priorities. 

■  Select  tail  numbers  of  aircraft  needed  for  contingency  operations. 

■  Other  coordinating  activities. 

The  Sortie  Generation  Flight  normally  consists  of  Aircraft,  Specialist,  and  Weapons  sections.  The 
flight  is  responsible  to  the  SMO  for  ensuring  sufficient  numbers  and  specialties  of  personnel  are 
available  to  support  the  production  effort.  Aircraft  sections  are  comprised  of  dedicated  crew 
chiefs,  assistant  crew  chiefs,  and  aircraft  technicians.  Aircraft  sections  are  the  primary  work 
centers  responsible  for  maintaining  the  assigned  aircraft.  Common  aircraft  section  tasks  are 
servicing  scheduled  and  unscheduled  maintenance,  pre-flights,  basic  post-flights,  home  station 
checks,  and  launch  and  recovery  of  aircraft.  The  Specialist  section  includes  dedicated 
technicians  that  perform  on-aircraft  repairs;  troubleshooting,  component  removal/replacement 
and  aircraft  ground  handling.  The  Weapons  section  performs  the  loading  and  maintenance  of 
weapons  onto  the  aircraft. 

The  Production  Superintendent  (PS)  is  also  resident  in  the  Sortie  Generation  Flight  and  is  the 
focal  point  and  coordinator  for  maintenance  production  decisions  within  the  squadron.  The  PS  is 
a  key  figure  in  achieving  sortie  generation  goals  and  directives  and  as  such  makes  daily 
decisions  that  enable  the  squadron  to  successfully  implement  AEF  directives  regarding 
deployment  of  forces,  personnel,  and  resources  to  employment  locations.  For  the  purpose  of  this 
thesis,  the  PS  functions  as  the  designated  decision-maker  for  the  individual  decision  task 
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examined  by  the  interactive  DSS.  ACCI  21-101  section  7.8  outlines  the  PS  responsibilities  as 
follows: 

■  Directs  the  maintenance  effort  using  resources  from  Sortie  Generation  and  Sortie 
Support  Flights. 

■  Is  involved  in  developing  and  implementing  the  Monthly  and  Weekly  Maintenance 
Plans,  ensuring  resources  are  available  to  meet  these  plans. 

■  Serves  as  the  squadron  point  of  contact  for  all  decisions  relating  to  squadron 
maintenance  production. 

■  Coordinates  with  other  squadron  PS  for  squadron  to  squadron  support. 

■  Attends  Daily  Maintenance  Meeting. 

■  Knows  the  status  of  assigned  aircraft. 

■  Knows  the  actions  required  under  unit  SIOP/  Contingency  Plans 

■  Directs  aircraft  generation  flow 

■  Coordinates  with  the  MOC  for  support  required  outside  their  control. 

■  Advises  the  MOC  of  conditions  that  may  disrupt  orderly,  controlled  execution  of  the 
Maintenance  Plan. 

In  this  section,  individuals  and  organizational  entities  were  discussed  to  provide  a  functional 
representation  of  the  organizational  environment  in  which  the  Production  Supervisor  (PS) 
conducts  business.  Utilizing  organizational  resources  to  gather,  update,  and  interact  with 
pertinent  resources;  the  PS  derives  images  that  correspond  to  Image  Theory  values,  trajectory, 
and  strategic  direction.  Directives  received  from  higher  organizational  components,  like  the 
Operations  Squadron  (OS)  through  the  Squadron  Maintenance  Officer  (SMO),  coupled  with  the 
sphere  of  authority  and  responsibility  outlined  by  pertinent  regulations  make  up  the  PS  non¬ 
physical  constraints  to  decision  making.  The  tangible  realities,  limiting  factors  of  personnel, 
resource  allocation,  facilities,  and  aircraft  limitations  produce  the  physical  constraints  to  the  sortie 
production  decision  space.  Production  Superintendents,  thus  constrained,  make  adoption 
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decisions  and  progress  decisions  using  appropriate  image  theoretical  tests  (Compatibility  and 
Profitability)  to  generate  aircraft  maintenance  production. 


3.2.2  Processes  and  Activities  (Sortie  Generation) 


The  process  and  activities  associated  with  sortie  generation  define  the  properties  necessary  and 
sufficient  for  control  of  physical  work  activities  and  use  of  aircraft  to  perform  the  mission. 
Discussed  in  this  section  is  the  iterative  process  of  coordinating  activities  between  the  various  OS 
components  to  achieve  the  organizational  objective.  This  section  examines  the  work 
environment  from  the  perspective  of  system  output  extracted  from  procedural  utilization  of  the 
previously  discussed  functional  elements  of  the  OS.  Significant  processes  and  activities  include, 
but  are  not  limited  to,  Aircraft  Scheduling,  Aircraft  Status  Reporting,  and  Flight  Line  Maintenance 
processes  (see  Figure  4).  These  processes  transfer  the  direction  from  higher  headquarters  into 
the  day-to-day  activity  of  a  flying  unit  and  provide  feedback  to  upper  management.  The  following 
paragraphs  depict  the  basic  processes  resulting  in  sustainable  sortie  production. 
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Flying  and  Maintenance  Scheduling  is  the  process  of  enacting  the  Flying  Hour  Allocation  program 
through  the  planning  cycles  of  operations  and  maintenance  quarterly,  monthly,  and  weekly 
schedules.  Flying  hour  goals  and  directives  are  distributed  to  the  individual  unit  through  the 
USAF  Major  Command  Headquarters.  These  directives  include  the  required  flying  hours, 
sorties/missions,  flying  days  in  each  month,  known  and  projected  special  mission  requirements, 
configuration/munitions  requirements,  and  aircraft/aircrew  alert  requirements.  These  parameters 
are  used  to  generate  annual  allocations  of  flying  hours  and  utilization  rates  to  the  individual  flying 
squadrons.  Principle  areas  of  concern  are  mission  accomplishment  and  improved  efficiency. 

Flying  squadron  input  is  used  to  generate  the  maintenance  planning  cycle.  The  maintenance 
planning  cycle  ensures  the  proper  and  effective  use  of  maintenance  resources.  Long  range 
planning  of  this  sort  is  needed  to  arrange  quarterly  flying  hour  programs,  TCTO  programs,  depot 
inspections,  phase  inspections,  scheduled  exercises,  and  deployments.  Scheduling  and  planning 
of  activity  results  in  general  operating  parameters  that  maintenance  assets  use  to  determine  the 
day-to-day  work  associated  with  the  unit.  These  measures  of  performance  and  measures  of 
effectiveness  represent  the  guiding  force  for  sortie  production  and  supervision  in  an  attempt  to 
limit  the  complexity  and  uncertainty  associated  with  logistical  processes. 

Limiting  uncertainty  does  not  guarantee  its  absence  and  supervisors  must  maintain  close  scrutiny 
on  operations  and  aircraft  status.  To  accomplish  necessary  feedback  the  Aircraft  Status  and 
Reporting  process  is  used.  Aircraft  status  is  depicted  using  a  code  assigned  to  a  specific  aircraft 
that  shows  its  mission  capability  rating.  The  current  state  of  aircraft  subsystems  is  compared  to 
the  Minimum  Equipment  Subsystems  List  (MESL)  to  determine  the  combat  capability  of  the 
asset.  The  MESL  lists  all  the  systems  and  subsystems  needed  for  full  mission  performance  and 
outlines  a  unit’s  specifically  assigned  wartime,  training,  and  test  missions  and  the  systems  and 
subsystems  that  must  be  working  for  a  unit  to  accomplish  those  missions.  Aircraft  are  given  a 
code  depicting  aircraft  status  as  follows: 
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■  Fully  Mission  Capable  (FMC):  All  systems,  subsystems,  and  components  listed  in 
the  Full  System  List  (FSL)  column  are  fully  functional. 

■  Partially  Mission  Capable  (PMC):  One  or  more  systems,  subsystems,  or 
components  are  not  working  and  the  aircraft  can  perform  some  but  not  all  of  its  basic 
missions. 

■  Non-Mission  Capable  (NMC):  Aircraft  is  incapable  of  performing  any  of  its  basic 
missions  as  outlined  in  the  MESL. 

Production  Superintendents  often  broaden  the  scope  of  aircraft  status  to  include  the  aircraft  state 
(configuration,  fuel,  weapons,  location,  etc.)  at  a  point  in  time.  This  definition  reflects  changes 
brought  on  by  the  Flight  Line  Maintenance  process,  but  is  a  less  formal  definition  and  is  not  part 
of  the  Aircraft  Status  and  Reporting  procedure. 

The  Flight  Line  Maintenance  process  is  designed  to  maintain  a  level  of  aircraft  readiness  at  some 
point  beyond  aircrew  and  operational  requirements  and  to  provide  combat  ready  aircraft  for  surge 
capacity  and  mobility  commitments.  Its  objective  is  to  provide  safe,  flyable  aircraft,  in  the  proper 
configuration,  when  and  where  needed  to  satisfy  aircrew  training  and  operational  mission 
requirements.  During  this  process,  maintenance  personnel  make  preparations  for  landing,  park 
the  aircraft,  debrief  the  aircrew,  perform  post-flight  inspections,  work  in-flight  discrepancies, 
schedule  maintenance  actions,  and  prepare  for  the  next  mission.  All  actions  are  documented  and 
are  posted  to  the  aircraft  information  system’s  database. 

In  this  section,  processes  are  defined  that  translate  directives  into  squadron  activities  (Aircraft 
Scheduling),  provide  feedback  and  accountability  in  the  form  of  documentation  (Aircraft 
Reporting),  and  produce  aircraft  missions  (Sortie  Production)  through  the  system  environment. 
Various  elements  of  each  of  these  processes  are  used  by  all  members  of  the  organization  to  elicit 
the  desired  response,  sustainable  sortie  generation.  The  next  section  describes  how  an 
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individual  decision  task  is  examined  to  determine  the  needs  of  the  decision-maker  within  the 


system  environment. 

3.2.3  Decision  Making  Activity  Analysis  (Deployment  Aircraft  Selection  Task) 

One  prototypical  production  task,  choosing  which  aircraft  should  be  deployed  as  a  part  of  an  AEF 
rotational  cycle,  was  chosen  from  the  core  Production  Superintendent’s  responsibilities  to 
demonstrate  the  decision-making  activity  endemic  to  the  Sortie  Generation  domain  under  the  new 
AEF  concept  of  operations.  The  deployment  of  tailored  Unit  Type  Codes  (UTC)  for  AEF 
deployment  is  novel  in  that  traditional  deployment  encompassed  the  entire  squadron  and  not  just 
partial  representation  of  the  unit.  Now,  six  aircraft  may  be  chosen  to  deploy,  whereas  previous 
deployments  included  all  mission  capable  aircraft  in  the  squadron.  Squadron  personnel  must 
prepare  and  perform  all  planning  activity  within  a  24-hour  period  after  notification. 

Interviews  identified  two  types  of  choice  strategies  dependent  upon  duration  and  utilization  of 
aircraft  while  on  deployment.  Choice  strategy  1  was  used  to  select  six  aircraft  for  deployment  if 
phase  inspections  were  not  expected  to  be  performed  at  the  contingency  location.  Subject  matter 
experts  utilized  this  strategy  if  the  expected  total  time  of  aircraft  flight  operations  was  100  hours  or 
less.  Choice  strategy  2  was  used  if  aircraft  utilization  at  the  deployment  location  suggested  that 
phase  inspections  would  be  performed  during  deployment.  Only  choice  strategy  1  was 
incorporated  into  the  decision  support  system  due  to  the  expectation  that  AEF  deployments 
would  be  rapid  response,  short  duration  actions  designed  to  expedite  operations  and  holding 
actions  before  the  arrival  of  more  permanent  forces.  The  emphasis  being  on  speed  and 
projection  of  force  instead  of  on  full-scale  large  deployments.  The  interactive  decision  support 
system  used  computer  suggestions  to  portray  the  choice  strategy  to  the  user. 

Eight  evaluative  subtasks  were  identified  for  choice  strategy  1  and  listed  in  Table  1.  These 
subtasks  were  analyzed  to  determine  the  rejection  threshold  and  the  variables  associated  with 
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the  profitability  tests.  Decision  tasks  served  to  first  eliminate  aircraft  from  consideration  that  did 
not  meet  specified  criteria  (Compatibility  Test)  and  second,  choose  the  best-suited  aircraft  based 
on  subjective  expected  utility  (Profitability  Test).  Compatibility  tests  performed  by  the  Production 
Superintendent  remove  aircraft  when  the  numbers  of  violations  in  aircraft  mission  capability, 
repair  history,  and  Phase  inspection  times  are  above  the  minimum  threshold  values  for  the 
strategy  being  used.  Profitability  tests  performed  include  critiquing  aircraft  based  on  scheduled 
maintenance,  outstanding  unscheduled  maintenance,  time  to  Phase  inspection,  aircraft 
configuration,  and  aircraft  location. 

During  the  mission  capability  or  aircraft  status  compatibility  test,  the  Production  Superintendent 
excludes  all  aircraft  from  consideration  that  are  designated  NMC  or  PMC  if  at  least  12  aircraft 
remain  to  pass  to  the  next  compatibility  test.  NMC  or  PMC  aircraft  are  not  likely  to  change  status 
within  the  24-hour  planning  deadline  called  for  by  AEF  requirements.  Aircraft  are  also  eliminated 
based  on  repair  history  considerations  associated  with  the  remaining  aircraft.  An  aircraft  that  has 
a  history  of  repeat/recurring  problems  may  not  be  able  to  successfully  execute  a  transatlantic 
crossing  or  fulfill  its  mission  once  at  the  employment  location.  Repeat/recurring  maintenance 
actions  that  have  no  known  explanation  do  not  effect  the  mission  capability  rating  of  an  aircraft  if 
the  problem  is  not  currently  present,  but  must  be  considered  likely  to  effect  wartime  efforts  if  the 
problem  is  in  a  critical  system.  This  characteristic  is  commonly  referred  to  as  the  aircraft’s 
‘personality’  and  denotes  the  increased  likelihood  of  specific  maintenance  actions  due  to  prior 
repairs  and  factory  defects.  The  complexity  and  interrelation  of  aircraft  subsystems  prohibits  the 
detection  and  repair  of  all  system  failures  and  must  not  be  ignored. 
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Evaluation 

Evaluation 

Order 

Decision  Type 

Test  Type 

Information 

Requirements 

Aircraft  Status 

i 

Adoption  Decision 

Compatibility  Test 

Mission  Capability  Rating,  Aircraft  History 

Phase  #1 

2 

Adoption  Decision 

Compatibility  Test 

Time  until  Phase  Inspection,  Phase  Month,  Conflicting 

Activities 

Scheduled  MX 

3 

Adoption  Decision 

Profitability  Test 

All  known  scheduled  maintenance  actions  for  the  period 

of  the  deployment 

Unscheduled  MX 

4 

Adoption  Decision 

Profitability  Test 

Outstanding  repair  actions,  time  to  complete  repairs, 

impact  of  repair  actions 

Phase  #2 

5 

Adoption  Decision 

Profitability  Test 

Time  to  Phase  Inspection 

Aircraft  Configuration 

6 

Adoption  Decision 

Current  configuration  of  aircraft 

Aircraft  Location 

7 

Adoption  Decision 

Profitability  Test 

Location  of  aircraft  relative  to  the  airfield 

Monitor  Goal  State 

Continuous 

Progress  Decision 

Compatibility  Test 

Mission  update 

Table  1 :  Identification  of  subtasks. 


The  Phase  Inspection  compatibility  test  takes  into  account  the  time  remaining  before  an  aircraft  is 
due  for  major  scrutiny  (300  hours  of  flight  time),  taking  it  off  the  list  of  available  assets.  If  an 
aircraft  comes  due  for  Phase  inspection  while  on  deployment,  the  aircraft  is  not  available  to  be 
used  for  combat  missions  and  must  be  held  until  maintenance  personnel  can  fit  the  inspection 
into  an  already  crowded  schedule.  Therefore  the  rejection  threshold  for  this  test  is  the  expected 
total  number  of  hours  aircraft  are  needed  to  fly  during  deployment.  If  aircraft  are  deployed  for  an 
extended  period  of  time  Phase  inspection  cannot  be  avoided,  but  every  effort  is  made  to  reduce 
the  workload  of  maintenance  personnel  while  on  deployment.  Phase  inspection  cycles  are 
generally  displayed  on  a  Phase  Flow  chart  depicting  in  descending  order  the  tail  number  of 
aircraft  by  time  remaining  until  inspection  is  due.  A  line  is  drawn  showing  the  ideal  graduated 
descent  enabling  aircraft  to  enter  Phase  incrementally.  Incremental  entry  into  Phase  enables 
sufficient  aircraft  to  remain  in  service  to  accomplish  mission  requirements.  Decision-makers,  at 
this  stage,  generally  eliminate  from  consideration  any  aircraft  that  is  due  for  Phase  inspection 
within  flight  time  allotted  for  the  deployment.  Remaining  aircraft  are  included  for  consideration  in 
the  next  stage  of  profitability  tests.  If  the  number  of  aircraft  surviving  the  two  compatibility  tests  is 
less  than  the  number  of  aircraft  needed  for  deployment  the  current  choice  strategy  is  abandoned 
and  a  new  strategy  adopted. 
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The  first  profitability  test  conducted  by  Production  Superintendents  is  a  review  of  the  importance, 
frequency,  and  density  of  scheduled  maintenance  items  to  be  performed  on  the  aircraft. 
Examples  of  common  aircraft  scheduled  maintenance  items  include  wash  and  corrosion  control 
checks,  10  hour  throttle  grip/flame  sensor  inspection,  50  hour  miniforce  check,  25  hour  borescope 
inspection,  and  50  hour  borescope  inspection  of  aircraft  blade  retainer.  Scheduled  inspections 
vary  between  aircraft  mission  design  series  and  are  determined  by  specifications  provided  by  the 
manufacturer.  Regular  inspections  enable  the  safe  and  reliable  operation  of  aircraft  and  enable 
long  service  life.  Aircraft  due  for  major  inspections  of  critical  systems  are  ranked  higher  in 
importance  than  inspections  involving  non-critical  systems.  Importance  of  aircraft  scheduled 
events  were  rated  by  subject  matter  experts  for  this  particular  task  to  reflect  the  impact  the  item 
would  have  on  deployment  sortie  generation.  Rated  items  were  categorized  into  three-color 
codes,  red  (High  Importance),  blue  (Medium  Importance)  and  green  (Low  Importance).  Time  to 
conduct  the  inspection  and  equipment  involved  in  the  process  are  also  major  considerations.  The 
decision-maker  utilizes  the  aircraft  schedule  provided  by  PS&D  section  to  make  the  assessment, 
rank  ordering  aircraft  from  best  to  worst. 

Unscheduled  maintenance  review  refers  to  the  evaluation  of  aircraft  based  on  broken,  cracked,  or 
out  of  limit  components  found  during  inspection  or  reported  by  pilots.  Aircraft  sent  on  deployment 
should  necessarily  be  as  free  of  problems  as  possible.  Therefore,  a  thorough  review  of 
outstanding  repairs  is  necessary.  Aircraft  are  ranked,  from  best  to  worst,  based  on  the  time  to 
repair  the  item  and  the  manpower  necessary  to  complete  the  work  order.  Repairs  to  non-mission 
essential  components,  regardless  of  the  time  and  manpower  needs,  are  ranked  lower  than 
mission  essential  component  repairs. 

Phase  inspection  times  are  re-evaluated  in  the  next  stage  to  determine  aircraft  that  have  more 
time  until  inspection  is  due.  Aircraft  with  that  will  not  come  due  for  inspection  within  the  time 
frame  of  the  deployment  are  ranked  higher  than  are  those  requiring  more  immediate  attention. 
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Aircraft  best  meeting  this  need  are  located  from  left  to  right  on  the  Phase  Flow  chart  and  are 
easily  identified.  Any  times  that  are  over  the  projected  deployment  time  satisfy  the  requirement, 
but  those  aircraft  with  greater  time  until  inspection  are  better  due  to  unforeseen  occurrences  that 
might  effect  the  duration  an  aircraft  remains  in  theater. 

Next,  aircraft  configuration  is  evaluated  to  determine  those  aircraft  meeting  the  mission 
configuration  requirement  before  deployment.  Configuration  refers  to  the  presence  or  lack  of 
armaments,  weapons,  or  specific  mission  systems  on  the  airframe.  For  example,  if  aircraft  are 
needed  for  overseas  deployment,  wing  tanks  are  used  to  extend  the  range  an  aircraft  can  fly.  If 
wing  tanks  are  already  present  on  the  airframe,  less  time  is  needed  to  configure  the  aircraft  for 
duty.  The  decision-maker  ranks  aircraft  based  on  the  time  and  effort  necessary  to  reconfigure 
aircraft  for  duty. 

Also,  aircraft  are  evaluated  to  determine  the  most  expedient  airfield  location  for  preflight  activities 
and  launch.  Aircraft  requiring  loading  of  armaments  must  comply  with  regulations  concerning 
distance  from  critical  facilities  and  other  assets  in  the  event  of  emergency.  Similarly,  aircraft 
requiring  fuel  need  to  be  away  from  sensitive  areas.  With  larger  airframes,  movement  on  the 
airfield  can  be  extremely  difficult  due  to  space  limitations  and  availability  of  equipment  used  in  the 
transfer.  Smaller  airframes,  like  the  F-16,  are  more  readily  accommodated.  Aircraft  are  rank- 
ordered  by  least  effort  to  comply  with  regulation  limits  during  preflight  and  launch  activities. 

Finally,  decision-makers  continually  evaluate  the  progressive  creation  of  the  available  aircraft  list. 
A  comparison  of  current  decision  progress  against  the  internal  image  constituents  (value, 
trajectory,  and  strategic  images)  impacts  the  determination  of  satisfactory  progress.  In  general, 
this  relates  to  the  trust  decision-makers  have  in  the  process  utilized  to  make  the  decision  and  the 
efficacy  of  the  projected  outcome.  Solution  sets  not  conforming  to  internal  measures  of  validity 
and  confidence  will  not  be  accepted  and  initiate  a  new  approach  to  the  problem. 
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Categorizing  the  identity,  decision  type,  and  test  type  of  subtasks  enable  the  development  of  a 
semi-structured  model-based  environment  for  implementation.  In  the  next  section,  decision 
subtasks  and  image  theoretical  characterizations  are  combined  to  form  a  knowledge-based 
model  of  aircraft  selection. 

3.3  Model  Implementation 

A  combination  pictorial  and  spreadsheet-based  interface  is  commonly  used  in  many  electronic 
information  systems  to  present  information  and  collect  user  input.  Users  consult  the 
spreadsheet-based  information  presented  by  scrolling  through  lists  of  information,  selecting 
control  nodes  leading  to  other  pages,  and  by  evaluating  forms  or  figures  displayed  on  the  screen. 
User  input  is  accomplished  by  inserting  aircraft  tail  numbers  into  an  exclusionary  list  for 
compatibility  tests  and  an  ordered  list  for  profitability  tests.  Implementation  of  Image  theoretical 
decision  structures  are  accommodated  by  utilizing  a  structured  interactive  approach  limiting  the 
user  to  the  specified  sequence  of  events  and  types  of  decisions,  but  not  limiting  the  input  within 
the  eight  identified  decision  subtasks.  Adoption  decisions  are  assisted  employing  user-input  text 
fields  to  either  exclude  aircraft  from  a  list  or  rank  aircraft  in  an  ordered  list.  Progress  decisions 
are  accommodated  using  a  feedback  mechanism  displaying  a  cumulative  rank  ordered  list  of 
aircraft  as  the  decision  process  proceeds.  Users  can  effectively  account  for  changes  to  internal 
image  states  by  overriding  computer  suggestions  at  any  point.  Figures  5-13  display  screen 
captures  of  the  individual  decision  screens  directly  corresponding  to  the  seven  discrete  decision 
subtasks.  Users  are  asked  to  perform  the  aircraft  selection  task  within  the  context  of  a 
deployment-based  scenario.  Scenario  instructions  are  listed  in  Table  2  and  displayed  to  the  user 
in  Figure  5. 
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Scenario:  You  are  the  Production  Superintendent  of  a  maintenance  squadron  that  provides 
maintenance  support  for  18,  F-16  block  50  aircraft.  You  have  been  tasked  by  your  supervisor 
to  assess  the  current  inventory  of  aircraft  in  your  squadron  and  decide  which  six  aircraft  should 
be  sent  to  support  an  AEF  deployment  to  Aljaber,  Kuwait.  Upon  arrival  at  the  deployment 
destination  the  aircraft  will  be  used  to  provide  100  hours  of  air-to-air  coverage  to  on-going 
operations  in  the  area. 

Use  the  program’s  step-by-step  instructions  and  the  information  provided  to  select  six  aircraft 
from  your  squadron  to  support  the  AEF  directive.  These  steps  will  guide  you  through  an  expert- 
derived  process  to  find  the  best  aircraft  in  the  squadron  to  send  on  deployment.  Any  questions 
should  be  directed  to  the  experimenter.  You  may  practice  as  many  times  as  necessary,  until 
you  feel  comfortable  enough  to  complete  the  task.  When  you  are  ready  to  attempt  a  timed  trial, 
inform  the  experimenter  and  he  will  help  you  begin.  Timing  of  the  task  begins  when  the  first 
screen  is  displayed.  Thank  you  for  your  participation. 


Table  2:  Scenario-based  Instructions. 


3.3.1  Compatibility  Tests 


The  first  identified  compatibility  test  is  based  on  aircraft  status  information.  Figure  6  displays  the 
information  provided  to  the  user  for  this  purpose.  Shown  on  screen  is  pertinent  data  necessary  to 
determine  the  true  mission  capability  of  all  squadron  aircraft.  Users  peruse  the  spreadsheet  and 
activate  the  aircraft  history  screen  by  left  clicking  on  the  individual  aircraft  repair  history  field. 
Once  aircraft  have  been  identified  that  do  not  meet  minimum  criteria,  the  user  selects  the  correct 
aircraft  tail  number  from  the  list  and  moves  it  to  the  box  provided  for  exclusion.  A  computer 
suggestion  box  is  provided  illustrating  the  model’s  selection  of  aircraft  for  exclusion.  Users  signify 
agreement  or  disagreement  by  entering  only  those  aircraft  tail  numbers  into  the  exclusion  field. 
The  process  progresses  to  the  next  screen  when  the  NEXT  button  is  selected.  Only  user-input 
values  are  incorporated  into  the  remainder  of  the  program  and  any  user-excluded  tail  numbers 
are  eliminated  from  consideration  for  the  duration  of  the  program. 
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Pfesse  read  the  following  scenatfe  and '^siructldns  cer£fy%.  : 

SCENARIO:  Yfeu  art?  the  production ':  supervisor  of  a  maintenance 
squadron  that  provides  maintenance  support  for  1Sf  F-t6  block  SO 
aircraft:  You  have  been  tosNed  ‘byywf  $j$asvi$hf  to  ass&$  $  ffii 
current  inventory  of  sirbrsh  in  your  squadron  and  decide  wfiibb  six : 
.aircraft  should  be  sent  to  support  an  AEF  depio>wsnf  to  ApCst, 

.  Kuwait,  Upon  arrival  at  Che  deplsymenf  destination  the  aircraft  will  be 
used  to  provide  too. hours  of  air-to-air  coverage  of  prvgorig  operations 
in  the  area  ■■ 

Ta^K:  UsetH«  program^  sfep-by-step  [natnjctib'ns  and  mejnibfmadon 
prodded  to  select  six  aircraft  from  ^ur  squadron  to  support  tee'ASF 
directive.  These  steps  wifi  quids  you  through  an  expert-derived  process 
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Figure  5:  DSS  Introductory  screen  displaying  instructions. 


1  I  rtf  r  McSsveP  SS 


Figure  6:  Aircraft  Status  screen. 

The  next  compatibility  test  involves  removal  of  aircraft  due  to  low  Phase  inspection  times.  Figure 
7  displays  the  screen  capture  for  this  subtask.  The  computer-generated  suggestion  removes 
aircraft  from  the  working  list  of  available  aircraft  if  the  time-to-Phase  value  is  less  than  40  hours. 
Users  may  signify  their  approval  or  disapproval  of  computer  suggestions  by  selecting  aircraft  tail 
numbers  from  the  spreadsheet  and  including  them  in  the  text  box  for  removal.  Aircraft  are 
removed  from  the  available  aircraft  list  when  the  NEXT  button  is  activated. 
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Figure  7:  Phase  Inspection  screen  #1. 

3.3.2  Profitability  Tests 

The  first  profitability  test  utilizes  information  concerning  schedule  maintenance  actions.  Figure  8 
displays  the  information  screen  for  this  subtask.  Users  compare  the  list  of  available  aircraft 
against  the  density  and  importance  of  scheduled  maintenance  items  over  the  duration  of  the 
deployment  period.  Items  are  color  coded  to  represent  various  importance  ratings  from 
mandatory  actions  (red)  to  less  important  actions  (green).  The  user  is  asked  to  rank  order  the 
aircraft  tail  numbers  from  best  to  worst  in  the  text  field  provided.  Data  entry  is  accomplished  by 
selecting  a  tail  number  from  the  list  and  moving  it  to  the  appropriate  column.  Computer 
suggestions  are  provided  in  the  left-most  column.  The  subtask  is  completed  when  the  user 
activates  the  NEXT  button. 
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Figure  8:  Scheduled  Maintenance  screen. 


Outstanding  unscheduled  maintenance  items  are  evaluated  by  utilizing  aircraft  history  records 
and  time-to-complete  projections.  Figure  9  displays  the  information  screen  for  this  subtask. 
Aircraft  are  rank-ordered  based  on  the  severity  of  the  outstanding  repair  action,  the  time-to- 
complete,  and  the  impact  repair  will  have  on  the  deployed  mission.  Users  are  tasked  to  provide 
an  ordered  list  common  to  all  profitability  test  subtasks.  The  subtask  ends  when  the  user 
activates  the  NEXT  button. 
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Figure  9:  Unscheduled  Maintenance  screen. 


Figure  10  displays  the  screen  utilized  for  the  second  Phase  inspection  evaluation.  Users  rank- 
order  aircraft  tail  numbers  based  on  the  time-to-Phase  values  provided  in  the  accompanying 
chart.  Aircraft  with  larger  times-to-Phase  are  generally  considered  better  than  those  aircraft  with 
less  time  due  before  major  inspection.  A  second  computer  generated  list  is  provided  on  this 
screen  representing  the  cumulative  rank  ordering  of  available  aircraft.  Previous  to  this  screen 
cumulative  progress  was  only  identified  by  inclusion  of  the  aircraft  for  consideration.  At  this  stage 
two  screens  (Scheduled  and  Unscheduled  Maintenance)  of  rank  ordered  data  are  available  to 
construct  the  cumulative  rank  ordering  of  aircraft.  Displaying  cumulative  feedback  of  on-going 
processes  fulfills  the  progress  decision  requirement  for  users  to  be  cognizant  of  advancing 
solutions.  Aircraft  tail  numbers  are  ordered  and  the  subtask  ends  when  the  user  activates  the 
NEXT  button. 
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Figure  10:  Phase  Inspection  screen  #2. 


The  final  two  subtasks,  Aircraft  Configuration  and  Aircraft  Location,  are  evaluated  in  the  next  two 
screens  shown  in  Figures  11  and  12.  Aircraft  configuration  and  location  data  is  evaluated  to 
determine  which  aircraft  require  the  most  time  and  effort  to  make  ready  for  departure.  Locations 
of  individual  aircraft  are  shown  in  relation  to  other  aircraft,  facilities,  and  known  hazards.  Aircraft 
are  rank-ordered  and  values  are  submitted  by  activating  the  NEXT  button.  Figure  13  displays  the 
cumulative  list  and  allows  the  user  a  final  time  to  alter  the  solution  set.  Baring  any  further 
alteration,  the  entire  task  is  complete  and  the  user  is  provided  with  a  list  of  aircraft  tail  numbers 
rank-ordered  from  best  to  worst  for  use  choosing  aircraft  for  the  specified  deployment. 
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Figure  1 1 :  Aircraft  Configuration  screen. 
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Figure  12:  Aircraft  Location  screen. 
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Figure  13:  Approve  Solution  screen. 


3.4  Summary 


In  this  chapter,  a  methodology  was  outlined  to  develop  a  knowledge-based  model  using  work 
domain  analysis  techniques  and  implemented  in  a  decision  support  framework  for  use  in  a 
maintenance  organization.  One  continuous  and  seven  discrete  subtasks  were  found 
corresponding  to  an  Image  theoretical  expression  of  a  Production  Superintendent’s  choice 
strategy.  Identification  and  elaboration  of  this  framework  led  to  the  representation  of  the  process 
by  means  of  adoption  and  progress  decisions.  Relevant  subtasks  were  further  categorized  by 
types  of  decision  tests  performed  and  were  adapted  to  support  the  overarching  decision  process. 
The  resulting  decision  support  system  will  be  tested  in  the  following  chapter. 
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4.0  Evaluation 


In  order  to  evaluate  the  DSS  generated  by  the  research  methodology,  a  real-time  database 
capability  was  needed  to  simulate  the  environmental  conditions  of  near-term  Air  Force  database 
systems.  Simulated  real-time  data  was  used  for  two  purposes,  realism  and  bias  prevention.  For 
the  purposes  of  this  thesis,  near-term  database  capabilities  are  assumed  to  exist.  Real-time 
database  systems  behave  differently  from  current  systems  in  that  they  provide  more  accurate, 
timely,  and  reliable  information.  Inclusion  of  the  simulated  real-time  database  removes  those 
tasks  from  consideration  that  are  performed  merely  to  compensate  for  limiting  factors  associated 
with  poor  database  capability.  Inclusion  of  the  simulated  data  also  serves  to  remove  prior 
knowledge  of  aircraft  history  from  hindering  evaluation  of  the  DSS.  Air  Force  maintenance 
personnel  are  a  highly  cohesive  and  integrated  work  force.  It  is  not  unreasonable  to  assume  that 
subjects  drawn  from  this  group  may  have  prior  knowledge  of  existing  aircraft  used  in  any  current 
squadron.  Personnel  are  moved  periodically  from  base  to  base,  operate  in  close  proximity  and 
coordination  with  other  maintenance  units,  and  communicate  freely  across  squadrons  to  solve 
complex  issues.  Creation  of  a  squadron  of  aircraft,  flown  in  simulation  for  several  months, 
effectively  generates  a  completely  new  set  of  aircraft  data. 

Major  features  of  the  air  base  simulation  infrastructure  include  a  random  number  generator,  a 
statistical  distribution  calculator,  an  event  calendar,  simulation  clock,  failure  generator,  and  main 
simulation  loop.  The  failure  generator  incorporated  major  inspection  cycles,  unscheduled  failures 
to  system  components,  aborted  sorties  due  to  weather  and  scheduling  conflicts,  and  delays  due 
to  part  unavailability,  manning  difficulties,  and  mission  restrictions.  The  assignment  of  the 
simulation  was  to  generate  as  many  sorties  as  possible  while  conforming  to  realistic  limitations. 
Typical  restrictions  included  a  five-day  flying  schedule,  and  a  10-hour  flying  day.  Aircraft  were 
flown  for  18  weeks  to  fully  develop  the  repair  histories  and  establish  a  “personality”  pattern  for 
each  individual  aircraft.  Generated  aircraft  datasets  were  eliminated  from  use  if  flight  time  was 
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low  and  unscheduled  maintenance  actions  were  not  uniformly  distributed.  The  simulation  shared 
major  components  with  the  JADIS  architecture  for  logistics  simulation  as  outlined  by  Narayanan, 
etal  (1997). 

Exploration  of  DSS  efficacy  consisted  of  drawing  comparisons  between  the  effectiveness  of  an 
Information-Presentation-Only  tool  and  the  Interactive  DSS  tool.  The  first  section  outlines  the 
rationale  behind  the  use  of  an  Information-Presentation-Only  tool.  The  remaining  sections 
discuss  experimental  procedures  for  evaluating  the  Interactive  DSS. 

4.1  Information-Presentation-Only  Tool 

Utilization  of  an  Information-Presentation-Only  (IPO)  tool  as  a  substitute  for  data  collection  in  an 
actual  aircraft  squadron  is  necessary  to  eliminate  confounding  variability  due  to  user  data 
collection  and  user  prior  knowledge.  Users  frequently  collect  aircraft  data  for  the  selection  task 
by  personal  communication  with  pertinent  organizational  elements  (LOCIS,  1995).  Collection 
times  may  vary  due  to  collection  method,  interfering  tasks,  process  deficiencies,  and  prior 
knowledge  of  aircraft  conditions.  It  becomes  increasingly  difficult  to  separate  task  performance 
from  non-essential  variability  in  actual  field  conditions.  For  these  reasons,  an  IPO  collection  tool 
was  constructed  to  provide  more  realistic  baseline  measurements. 

The  IPO  tool  consists  of  a  series  of  informational  screens  with  an  accompanying  ‘scratch  pad’ 
that  serves  as  an  external  memory  storage  device.  The  scratch  pad  is  a  blank  list,  ordered  from 
one  to  eighteen.  This  list  is  external  to  the  informational  screens  in  that  it  is  always  visually 
present  as  the  user  navigates  through  the  screens.  The  user  interacts  with  the  information 
screens  by  freely  selecting  the  tabs  associated  with  each  of  the  screens.  Users  annotate  aircraft 
tail  numbers  to  the  scratch  pad  by  selecting  an  individual  tail  number  from  the  information  screen 
and  assigning  it  to  a  list  position  on  the  scratch  pad.  Users  navigate  through  the  information 
compiling  and  amending  the  aircraft  list  and  finish  the  task  by  selecting  the  END  button.  IPO 
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screens  are  comprised  of  Phase  Info,  Aircraft  Status,  Scheduled  Maintenance,  and  Location  of 
Jets. 


The  Phase  Info  screen  depicted  in  Figure  14  utilizes  the  standard  Phase  Flow  diagram  and 
typical  Phase-related  data  generally  employed  by  maintenance  personnel.  Data  includes  aircraft 
tail  number,  number  of  hours  until  Phase  inspection  is  due,  projected  Phase  month  aircraft  should 
become  unavailable  due  to  Phase,  fleet  time  for  the  entire  squadron,  and  annotated  remarks 
concerning  special  aircraft  scheduled  deployments,  exercises,  or  training. 


Figure  14:  IPO  Phase  Info  information  screen. 


The  Aircraft  Status  screen  shown  in  Figure  15  combines  elements  of  four  categories  of 
information:  status,  configuration,  aircraft  history,  and  unscheduled  maintenance.  Status,  as 
discussed  in  the  previous  chapter,  relates  to  the  mission  capability  of  an  individual  aircraft.  Since 
it  is  assumed  that  real-time  data  is  available,  status  refers  to  the  actual  condition  of  the  combat 
capability  of  the  aircraft  and  not  the  frequently  inaccurate  listed  condition  due  to  reporting  latency. 
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Similarly,  aircraft  configuration,  history,  and  unscheduled  maintenance  items  accurately  reflect 
current  aircraft  state. 
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Figure  15:  IPO  Aircraft  Status  information  screen. 


The  Scheduled  Maintenance  information  screen  depicted  in  Figure  16  allows  the  user  to  view 
aircraft  maintenance  schedules  in  Gantt  chart  format  for  a  period  of  one  quarter  year  of  squadron 
operation.  The  chart  is  viewed  by  using  the  scroll  bar  to  access  out-of-limit  representations  in  the 
viewing  area.  Identical  to  the  Interactive  DSS,  the  chart  is  color  coded  to  represent  importance 
rating  for  varying  levels  of  scheduled  inspections  and  processes.  Red  indicates  important 
inspections  possibly  interfering  with  aircraft  use  in  deployment  operations.  Yellow  indicates 
moderate  task  importance.  Green  indicates  scheduled  tasks  that  can  possibly  be  delayed  or 
have  limited  impact  on  an  aircraft’s  deployment  availability. 
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Figure  16:  IPO  scheduled  maintenance  screen. 
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Figure  17:  IPO  Location  of  Jets  information  screen. 


Figure  17  displays  the  last  screen,  Location  of  Jets.  This  screen  is  consulted  to  determine  aircraft 
location  relative  to  pertinent  airfield  features.  Aircraft  locations  are  depicted  in  green  with  tail 
numbers  listed  in  red.  Aircraft  in  flight  are  listed  in  a  special  section  of  the  display  as  well  as 
aircraft  residing  in  one  of  the  hangers. 

When  all  aircraft  have  been  assigned  to  a  location  on  the  scratch  pad’s  ordered  list  and  the  END 
TRIAL  button  is  activated,  data  collection  is  complete  and  the  time-to-complete-task  and  ordered 
list  are  saved  to  a  text  file  for  later  examination.  Differences  between  the  two  presentation 
methods,  IPO  and  IDSS  are  highlighted  in  Table  3  and  show  the  inclusion  or  exclusion  of  the 
identified  image  theoretical  constructs  identified  in  the  choice  strategy. 
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IDSS 

IPO 

Compatibility  Test 

Aircraft  Status 

Suggested  removal  of  aircraft  if: 

•  FMC  status  is  violated 

Phase  Inspection  #1 

Color  coding  of  deviations  from  standard 

Suggested  removal  of  aircraft  if: 

•  Time  until  Phase  is  less  than 

expected  deployment  duration 

Color  coding  of  deviations  from 

standard 

Profitability  Test 

Scheduled  Maintenance 

Color  coding  of  importance  rating 

Aircraft  ranked  based  upon: 

I(importance  rating)x(duration)x  (frequency) 

Color  coding  of  importance  rating 

Unscheduled  Maintenance 

Aircraft  ranked  based  upon: 

•  Lowest  time  to  complete  repairs 

Phase  Inspection  #2 

Color  coding  of  deviations  from  standard 

Aircraft  ranked  based  upon 

•  Highest  time  until  due  for  phase 

inspection 

Color  coding  of  deviations  from 

standard 

Configuration 

Aircraft  ranked  based  upon: 

•  Lowest  time  to  achieve  desired 

configuration 

Location 

Graphical  map  of  airfield  with  positions  listed 

Aircraft  ranked  based  upon: 

•  Lowest  time  to  achieve  desired 

location  on  ramp 

Graphical  map  of  airfield  with 

positions  listed 

Process  Constraints 

Enforced  step-by-step  process 

None 

Table  2:  Procedural,  graphical,  and  computer-aided  suggestions  for  each  presentation  method. 


4.2  Experimental  Design 
4.2.1  Hypothesis 

There  are  three  hypotheses  evaluated  in  this  experiment.  First,  the  IDSS  presentation  method 
provides  better  performance  environment  than  the  IPO  presentation  method.  Second,  the  IDSS 
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presentation  method  yields  a  faster  time  to  complete  the  task  than  does  the  IPO  presentation 
method.  Thirdly,  the  IDSS  presentation  method  produces  a  higher  subjective  confidence  in  the 
generated  solution. 

4.2.2  Variables 

Independent  variables  include  the  presentation  method  (IDSS  and  IPO)  and  the  aircraft  data  set 
(D3  and  D4).  Associated  with  these  variables  are  three  dependent  variables:  performance,  time- 
to-complete,  and  user  confidence. 

4.2.3  Equipment 

The  experiment  was  administered  at  the  445th  Air  Force  Reserve  Generation  Flight  and  the  Air 
Force  Material  Command  Headquarters  building,  both  located  at  Wright-Patterson  AFB  in 
Dayton,  Ohio  using  a  Micron  Pentium  II  laptop  computer.  A  15  inch  integrated  color  monitor 
attached  to  the  laptop  was  used  at  1024x768  pixels.  The  Interactive  DSS  and  IPO  test  conditions 
were  written  in  Java  version  1 .2,  and  executed  in  a  Windows  98  environment.  Program  input  was 
accomplished  using  a  computer  mouse  and  keyboard. 

4.2.3  Subjects 

The  subject  pool  for  this  experiment  consisted  of  maintenance  personnel  from  the  445th  Air  Force 
Reserve  Squadron  and  Air  Force  Material  Command  familiar  with  elements  of  the  selection  task. 
All  12  subjects  were  military  AFRES  or  active  duty  personnel  with  over  10  years  flight  line 
experience.  No  subject  had  any  physical  impairment  that  might  degrade  performance  with  a 
color  monitor,  keyboard,  or  mouse.  No  subject  was  compensated  for  his  or  her  time.  Subjects 
consisted  of  one  female  and  1 1  male  personnel. 
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4.2.5  Design  and  Procedure 

Subjects  participated  in  a  two-factor  crossed  experiment  (Presentation  Method  and  Aircraft  Data 
Set)  with  two  treatments  in  each  factor.  Repeated  measures  were  used  on  one  factor 
(Presentation  Method).  Order  of  presentation  for  the  two  factors  utilized  a  randomized  design.  In 
this  experiment  the  Interactive  DSS  and  IPO  were  evaluated  by  comparing  three  criteria,  time-to- 
complete,  percent  of  ordered  responses  that  correctly  identified  the  “ideal”  rank  order  established 
by  a  committee  of  experts,  and  subjective  confidence  ratings.  See  Table  3  for  design 
clarification. 


Presentation  Order 

Subject 

First 

Second 

i 

Dataset  3,  IDSS 

Dataset  4,  IPO 

2 

Dataset  3,  IDSS 

Dataset  4,  IPO 

3 

Dataset  3,  IDSS 

Dataset  4,  IPO 

4 

Dataset  3,  IPO 

Dataset  4,  IDSS 

5 

Dataset  3,  IPO 

Dataset  4,  IDSS 

6 

Dataset  3,  IPO 

Dataset  4,  IDSS 

7 

Dataset  4,  IDSS 

Dataset  3,  IPO 

8 

Dataset  4 ,  IDSS 

Dataset  3,  IPO 

9 

Dataset  4,  IDSS 

Dataset  3,  IPO 

10 

Dataset  4,  IPO 

Dataset  3,  IDSS 

11 

Dataset  4,  IPO 

Dataset  3,  IDSS 

12 

Dataset  4,  IPO 

Dataset  3,  IDSS 

Table  4:  Presentation  order  of  treatments  to  subjects  before  randomization. 


The  experiment  consisted  of  four  phases:  experiment  introduction,  presentation  method  practice 
trials,  data  collection  trials,  and  post-experimental  questionnaire.  Experiment  introduction  was 
used  to  obtain  the  participant’s  signature  on  the  consent  form  (see  Appendix  A),  acclimate  the 
subject  to  the  testing  environment,  and  a  written  explanation  of  the  task  and  scenario.  Practice 
trials  were  conducted  for  each  of  the  two  presentation  methods  using  a  practice  aircraft  dataset. 
Subjects  were  allowed  to  practice  until  comfortable  with  the  task  and  experimental  environment. 
After  training,  subjects  began  the  data  collection  trials  presenting  the  experimental  factors  using 
counter-balanced  design  in  Table  3.  At  the  conclusion  of  the  data  collection  trials,  subjects  were 
asked  to  fill  out  the  post-experimental  questionnaire  (see  Appendix  B). 
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4.3  Evaluation  of  Performance 


An  independent  committee  of  three  maintenance  Production  Superintendents  from  the  F-16 
System  Program  Office  at  Wright-Patterson  AFB  rank  ordered  subject  raw  position  scores  to 
determine  a  relative  performance  ranking.  Committee  members  performed  the  aircraft  selection 
task  using  the  IPO  for  each  aircraft  dataset  and  used  their  responses  as  the  ideal  rank  order  of 
aircraft  within  the  dataset  (See  Appendix  D).  After  comparing  the  respondent’s  rank  ordered 
performance  list  to  the  rank  ordered  list  of  the  committee,  a  relative  ranking  of  subject  responses 
was  derived.  Relative  rank  orders  were  listed  from  one  to  twenty-four  (number  of  subjects 
multiplied  by  two  presentation  conditions  a  piece),  with  one  representing  the  best  matching 
solution  to  the  committee  and  twenty-four  representing  the  least  correct  solution  set. 

4.4  Evaluation  of  Time  to  Complete  the  Task 

Times  to  complete  the  task  were  evaluated  using  a  One-Way  ANOVA  to  test  the  hypothesis  that 
the  mean  time  using  the  IDSS  was  less  than  that  for  the  IPO  condition,  or  Hi:  Pipo(d>  >  Pidss(D). 
where  p  is  the  mean  time  to  complete  the  task  per  each  condition  and  D  represents  the  aircraft 
dataset  used  during  the  trial. 

4.5  Evaluation  of  Post-Experimental  Questionnaire 

Post-experimental  questionnaire  responses  were  categorized  by  experience,  subjective 
confidence  rating,  and  write-in  comments.  Experience  items  reflected  whether  or  not  a  subject 
had  previous  familiarity  with  individual  aspects  of  the  aircraft  selection  task.  Responses  were 
used  to  determine  the  level  to  which  subjects  could  comprehend  and  express  knowledgeable 
opinions  on  the  information  format  and  content  of  the  presentation  method.  Subjective 
confidence  ratings  were  collected  using  a  series  of  fourteen  questions  eliciting  preferences  about 
assurance  in  the  generated  solutions.  Each  question  used  a  seven-point  scale  to  determine  the 
level  to  which  the  presentation  method  better  expressed  the  users  point  of  view.  Write-in 
comments  were  expressed  in  the  blank  spaces  provided. 
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5.0  Results 


5.1  Performance  Analysis 

Performance  data  from  the  experiment  were  assessed  using  the  non-parametric  Kruskal-Wallis 
Test  to  evaluate  the  hypotheses,  H0:  Pidss  =  Pipo,  where  p  indicates  the  mean  rank  ordered 
response  of  all  subjects  using  the  designated  presentation  method.  The  Kruskal-Wallis  test 
statistic  (K=.021332)  proved  to  be  less  than  the  X2(.0i,i)Of  6.637.  Therefore,  H0  cannot  be 
rejected  indicating  no  significant  difference  between  the  subject  IDSS  solutions  and  the  subject 
IPO  solutions  at  an  alpha  of  .01 . 

5.2  Time  to  Complete  Analysis 

Using  a  One-Way  ANOVA,  analysis  of  time  to  complete  data  indicate  that  when  presented  first, 
the  IDSS  condition  significantly  (1-tail  significance  =  .048)  improved  timed  performance  at  an 
alpha  of  .05.  Mean  difference  in  time  to  complete  the  task  was  670.67  seconds.  When 
presented  second,  the  IDSS  did  not  show  a  significant  difference  in  time  to  complete,  but  did 
show  practical  improvement  by  112.5  seconds  over  the  IPO  condition.  Further  analysis  using 
One-Sample  T-Tests  for  IDSS  first  presentation  order  and  IDSS  second  presentation  order  did 
not  indicate  significant  differences  between  the  two  test  conditions  for  either  criterion,  but 
generally  indicated  that  the  IDSS  took  less  time  with  mean  -257  seconds  and  mean  =  -527 
seconds  respectively. 

5.3  Subjective  Questionnaire  Analysis 

Subjective  questionnaires  were  utilized  to  examine  the  experience  level  of  subjects,  perception  of 
confidence,  and  written  comments.  Subject  experience  level  is  depicted  in  Table  7  and  indicates 
a  strong  majority  felt  they  had  experience  with  five  of  the  eight  areas  (strong  =  80%  or  above), 
and  a  simple  majority  stated  experience  in  seven  of  the  eight  areas.  Areas  listed  in  this  section 
pertain  to  major  maintenance  activities  relating  to  the  experimental  task  or  are  skill  items  that  aid 
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in  the  use  of  maintenance  computer  products.  It  is  unclear,  however,  how  all  subjects  can  be 
experienced  in  the  use  a  computer  program  (Question  6)  when  not  all  subject  have  used  a 
computer  to  process  information  (Question  5). 


Experience  and  Familiarity  with: 

Subjects  Responding  “Yes” 

(%) 

1 .  Choosing  aircraft  for  deployment  or  special 

58.3 

duty 

2.  Scheduling  aircraft  for  repair  and  inspection 

41.6 

3.  Determining  priority  of  repair  actions 

66.6 

4.  Serving  as  part  of  a  deployed  maintenance 

91.6 

unit 

5.  Using  a  computer  to  process  information 

91.6 

6.  Using  a  CAMS-like  computer  program  to 

100 

keep  track  of  aircraft  data 

7.  Viewing  and  assessing  the  impact  of 

83.3 

aircraft  repair  histories 

8.  Determining  if  an  aircraft  is  combat  ready 

91.6 

Table  5:  Subjective  responses  to  experience  questions 


Subjective  confidence  responses  were  used  to  derive  a  measure  of  inter-rater  reliability  using  the 
Coefficient  Alpha  test.  The  test  indicated  an  appropriately  level  of  internal  consistency  (.9697)  to 
use  the  questionnaire.  Reliability  Analysis  is  listed  in  Table  8. 
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Mean 

Std.  Dev. 

Cases 

Alpha  if  item 
deleted 

Q16 

4.1667 

1.9924 

12 

.9616 

Q17 

4.25 

2.1794 

12 

.9609 

Q18 

4.0 

2.0 

12 

.9618 

4.25 

2.3404 

12 

.9645 

Q20 

5.0 

2.1742 

12 

.9651 

Q21 

4.5 

2.4309 

12 

.9645 

Q22 

2.1667 

1.1146 

12 

.9747 

Q29 

3.25 

1.3568 

12 

.9686 

Coefficient 

Alpha 

.9697 

Table  6:  Reliability  Analysis  for  Confidence  Related  Questions. 


User  responses  to  the  subjective  confidence  portion  of  the  questionnaire  indicate  no  preference 
for  one  presentation  method  over  another  (mean  =  3.833,  with  4.0  being  a  neutral  response). 
Only  one  question  was  individually  suggestive  (Q22),  indicating  that  the  IPO  condition  allowed 
greater  freedom  to  explore  the  information  thoroughly.  All  other  response  means  were  centrally 
distributed  and  did  not  strongly  indicate  confidence  in  one  presentation  method  over  another. 

Written  comments  typically  involved  three  areas,  the  scenario,  the  display  format,  and  substantive 
information  related  material.  Written  comments  are  listed  in  Appendix  C  and  were  generally 
positive  in  nature.  Comments  generally  favored  the  concept  of  a  decision  support  system  for 
maintenance  especially  as  a  loose  association  of  relevant  informational  components,  or  as  a 
training  aid. 

5.4  Discussion 

The  first  hypothesis  for  the  experiment  was  that  the  mean  expert  committee  rating  for  the  IDSS 
condition  is  significantly  better  than  the  equivalent  statistic  for  the  IPO  condition.  This  would 
imply  that  a  better  decision  is  made  with  the  use  a  decision  support  using  an  image  theoretic 
system  constraining  the  user  so  that  each  identified  subgoal  is  considered  in  a  manner  more 
closely  related  to  user  image  states,  adoption  criterion,  and  expert  framing.  The  second 
hypothesis  stated  that  the  IDSS  condition  would  allow  users  to  accomplish  the  task  significantly 
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faster  than  the  IPO  condition.  Thirdly,  the  IDSS  presentation  method  produces  a  higher 
subjective  confidence  in  the  generated  solution.  The  following  subsections  discuss  the 
hypotheses  and  the  independent  variables. 

5.4.1  Performance  Hypothesis 

The  performance  hypothesis  is  not  supported  by  the  research.  The  results  show  that  the  IDSS 
produced  a  committee  ranking  that  was  not  significantly  different  than  did  the  IPO.  Verbal  and 
written  responses  on  the  post-questionnaire  indicate  that  lack  of  confidence  in  the  solution 
process  as  well  as  interface  problems  may  have  constrained  the  ability  of  users  to  interact 
effectively  within  the  system.  While  most  respondents  liked  the  computer  suggested  input,  it  may 
be  the  case  that  the  method  for  generating  computer  suggestions  be  made  visible  to  the  user. 

5.4.2  Time  to  Complete  Hypothesis 

The  time  to  complete  hypothesis  was  supported  for  the  first  order  of  presentation.  IDSS  Subjects 
completed  the  task  on  average  38%  more  quickly  than  the  IPO  condition.  Both  presentation 
methods  during  the  second  testing  session  tended  to  have  more  lengthy  times  to  complete.  This 
may  have  been  due  to  fatigue  or  other  confounding  factors.  There  was  evidence  that  would 
suggest  shorter  IDSS  completion  times  for  the  second  presentation  period,  but  the  differences 
proved  not  to  be  statistically  significant.  It  may  be  the  case  that  the  difference  in  completion  times 
for  the  second  testing  period  were  of  practical  significance  differing  pn  average  8.3%. 

5.4.3  Confidence  Rating 

The  hypothesis  involving  the  confidence  of  users  towards  the  generated  solution  was  not 
supported  by  the  research.  Subjects  proved  to  be  indifferent  in  terms  of  confidence  towards 
either  the  presentation  method  or  the  solutions  generated.  Mean  response  indicated  3.8  on  a 
seven-point  scale  with  one  indicating  strong  IPO  confidence  and  seven  indicating  strong  IDSS 
confidence.  Subjects  expressed  reluctance  to  commit  to  a  firm  opinion  of  confidence  due  to 

concern  that  the  decision  support  system  was  not  in  its  final  version,  interface  problems 
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decreased  the  ability  to  fully  explore  the  system,  and  aircraft  data  and  scheduled  maintenance 
items  were  not  detailed  enough  to  permit  precise  evaluation  during  the  task.  Though  subjects 
detailed  their  partiality  to  decision  support  features  such  as  computer  suggestions  and  guidance, 
color  coding,  and  desktop  convenience,  these  favorable  responses  did  not  reflect  in  confidence 
ratings. 
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6.0  Conclusions 


6.1  Contributions  of  the  Research 

This  thesis  made  practical  contributions  to  information  systems  research  by  outlining  a 
methodology  for  creating  decision  support  systems  in  complex  logistics  planning.  Contributions 
include  uncovering  decision  support  needs  for  the  organizational  strata  of  front  line  supervisors, 
applying  current  naturalistic  decision  theory  to  the  logistics  arena,  and  defining  a  level  of 
interaction  between  the  decision  maker  and  the  decision  support  system  that  accommodates 
subgoal  variation  while  maintaining  the  structure  of  the  knowledge-based  framework. 

Use  of  Image  theory  to  identify  decision  activity  greatly  increased  the  level  of  detail  and 
understanding  of  the  decision  process  of  Production  Superintendents.  Standard  methods  for 
identifying  decision  subtasks  do  not  necessarily  direct  the  method  in  which  those  tasks  will  be 
incorporated  into  the  decision  support  system.  Uncovering  the  image  states  of  the  subject  matter 
experts  led  to  the  acquisition  of  two  types  of  decision  strategies.  These  decision  strategies,  once 
identified,  not  only  changed  the  order  of  presentation,  but  highlighted  significant  structural 
changes  to  the  algorithms  used  in  providing  computer-aided  suggestions  to  the  user. 

Decision  support  needs  for  line  supervisors  in  logistics  have  been  largely  deferred  in  favor  of  high 
visibility  management  where  the  informational  needs  are  more  globally  oriented,  integrating  vast 
amounts  of  data  combined  with  uncertainty  and  a  heterogeneous  perspective  (Vicente,  1999). 
Lower  levels  of  management  have  different  needs  for  decision  support  than  upper  level 
managers,  but  remain  just  as  dependent  on  informational  computer  support.  Large  problem 
spaces,  even  at  the  squadron  level  complicate  the  thorough  examination  of  data  and  hinder  the 
process  of  shortfall  identification,  repeat  or  recurring  problem  analysis,  and  determination  of 
system  patterns  on  which  quality  decision  making  depends.  Impactful  decision  support 
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incorporating  satisficing  heuristics  may  enable  a  more  direct  and  immediate  application  of 
supervision  on  the  production  of  aircraft  sortie  generation  and  on  overall  squadron  production. 

Application  of  current  naturalistic  decision  theory  to  the  logistics  arena  provides  a  human- 
centered  perspective  that  takes  advantage  of  the  organic  teleological  processes  inherent  in 
human  mental  schemas.  Providing  information  constrained  to  fit  these  processes  allow  decision 
makers  more  direct  application  of  pertinent  information  to  affect  the  generated  solution  within  the 
context  of  the  knowledge-based  framework.  Formalization  of  the  decision  makers  natural  choice 
strategies  to  evaluate  decisions  based  on  compatibility  and  profitability,  adoption  or  progression, 
abbreviates  the  decision  process  reducing  internal  complexity  and  confusion,  thereby  reducing 
decision  time. 

The  third  contribution  of  the  research  is  the  further  clarification  of  the  role  of  humans  in  the 
human-machine  system.  By  defining  a  level  of  interaction  between  the  user  and  the  decision 
support  system,  image  states  and  goal  directed  behavior  inherent  to  logistics  organizations  can 
be  applied  to  the  decision  process  while  taking  advantage  of  computer  processing  speed  to 
identify  patterns,  process  heuristics,  and  make  computations.  Subgoals  can  be  evaluated 
visually,  using  the  computer  to  display  graphical  representations  of  information  patterns  while 
leaving  the  option  to  examine  the  data  individually.  Progress  towards  a  realistic,  useful  generated 
solution  can  be  monitored,  assessed,  and  altered,  allowing  the  decision  maker  to  review  the 
solution’s  compatibility  with  projected  needs. 

In  the  experiment,  the  Interactive  Decision  Support  System  (IDSS)  suggested  that  greater  speed 
can  be  realized  in  the  decision  process.  This  thesis  anticipated  an  increase  in  performance, 
confidence,  and  trust  in  the  generated  solution,  which  did  not  materialize  in  the  study.  Due  to  the 
low  number  of  participants,  only  general  conclusions  can  be  drawn  from  the  research  to  include  a 
suggested  increase  in  solution  generation  time  using  the  decision  support  system. 
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6.2  Limitations  of  the  Study 

There  were  several  limitations  of  the  study  that  included  small  size  and  experience  variation  of 
subject  population,  lack  of  expected  user-input  capability,  display  format  issues,  and  dissimilar 
aircraft  datasets.  These  limitations  in  design  represented  a  significant  stumbling  block  for 
participants  in  the  study  and  for  meaningful  evaluation  of  the  data. 

First,  the  number  of  subjects  in  the  experiment  needs  to  be  increased  for  a  more  statistically 
significant  relationship  to  exist.  Although  statistically  significant  results  were  found  at  an  alpha  of 
.05  for  time  to  complete  analysis,  the  low  number  of  subjects  would  suggest  that  an  alpha  of  .01 
would  strengthen  any  argument  for  the  decision  support  system.  Also,  it  is  possible  that  the 
mixture  of  active  duty  and  reserve  component  personnel  were  too  dissimilar  to  be  compared 
easily.  Reserve  personnel  largely  incorporate  ex-active  duty  personnel,  and  as  such  show  a 
great  increase  in  the  total  time  serving  in  a  maintenance  specialty.  While  at  first  glance  this  factor 
suggests  that  the  more  experience  in  maintenance  the  better,  it  may  not  be  true.  Unfortunately, 
significant  differences  occur  in  the  practice  of  maintenance  between  the  two  groups.  Active  duty 
personnel  move  from  base  to  base  over  the  course  of  their  career  and  do  not  serve  a  single 
squadron  for  very  long.  Reserve  personnel  serve  in  the  same  squadron  for  long  periods  of  time 
and  do  not  experience  the  problems  associated  with  moving.  It  is  possible  that  reserve  personnel 
do  not  have  enough  experience  assessing  a  group  of  aircraft  that  is  new  to  them.  The  low 
turnover  of  aircraft  and  personnel  in  Reserve  squadrons  implies  a  decrease  in  the  ability  to 
articulate  and  thoroughly  evaluate  aircraft  history  data. 

Second,  lack  of  program  functionality  that  the  majority  of  computer  users  have  come  to  rely  upon 
was  missing  or  ineffective  in  the  IDSS  and  IPO  programs.  Such  functions  as  “point-and-click” 
and  “drag-and-drop”  were  not  available  to  the  user.  Users  adapted  to  the  input  method  with  a 
significant  amount  of  complaint  using  a  modified  method  of  selection  of  aircraft  tail  number  and 
selection  of  position  reference  to  assign  tail  numbers  to  the  appropriate  positions.  Assignment  to 
the  scratch  pad  was  a  little  tricky  and  required  some  skill  to  use.  Practice  with  the  system 
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modified  this  factor,  but  remained  a  difficulty  throughout  the  experiment.  Also,  problematic  were 
the  occasional  overlapping  of  data  on  the  aircraft  location  diagram.  Overlapping  tail  numbers, 
while  not  prohibitive,  possibly  added  time  to  the  search  for  tail  numbers  of  interest,  especially  if 
the  user  did  not  expand  the  display  to  its  full  size.  Subjects  were  also  unhappy  that  they  needed 
to  scroll  through  data  to  view  the  contents.  Frequent  remarks  were  made  about  the  display 
graphics  not  fitting  in  their  entirety  on  the  display  surface. 

Lastly,  a  more  thorough  examination  of  the  aircraft  datasets  was  warranted.  The  time-to- 
complete  data  and  the  performance  data  suggest  that  the  two  datasets  may  have  been  unequal 
in  difficulty.  An  aircraft  dataset  that  had  a  greater  number  of  aircraft  that  were  mission  ready  and 
capable  of  deployment  would  add  to  the  difficulty  of  selecting  between  them.  Future  evaluations 
should  include  datasets  that  more  closely  resembled  one  another  in  their  complexity  level. 
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APPENDIX  A 


Informed  Consent  Document 

Interactive  Maintenance  Decision  Support  System 

This  signed  consent  is  to  certify  my  willingness  to  participate  in  this  research  study.  The 
purpose  of  this  study  is  to  determine  the  effectiveness  of  an  interactive  decision  support  system 
based  on  expert  knowledge-based  framework.  The  tasks  are  structured  to  emulate  the  way  an 
expert  may  perform  selection  of  aircraft  to  be  included  on  a  deployment.  I  am  being  asked  to 
participate  as  a  maintenance  user  of  the  decision  support  system,  selecting  aircraft  from  my  unit  to 
be  used  in  a  combined  AEF  deployment  force,  with  little  or  no  prior  knowledge  required  of  the 
system. 

I  will  be  trained  to  use  the  decision  support  system,  and  how  to  use  the  interface  to 
support  successful  completion  of  the  tasks.  I  understand  that  I  must  complete  a  practice  trial  in 
order  to  begin  the  actual  experimental  trials.  After  training  I  will  be  asked  to  make  decisions  based 
on  information  provided  by  the  computer  information  system.  The  average  time  required  will  be 
between  30  minutes  to  1  hour. 

During  the  qualifying  test  and  the  experimental  trials  I  will  be  using  a  Windows  based 
interface  to  seek  the  information  mentioned  in  the  tasks.  I  understand  that  I  must  have  experience 
using  a  mouse  and  keyboard,  be  an  experienced  maintenance  professional,  and  have  experience 
with  the  CAMS  system  to  participate,  and  I  will  be  trained  in  ail  other  aspects  of  the  task. 

After  performing  the  selection  task,  I  will  be  asked  to  fill  out  a  questionnaire  (Approx  10 
minutes  long)  that  asks  me  to  rate  relevant  items  about  the  display  and  model. 

The  results  of  this  study  may  be  used  to  help  determine  the  efficiency  of  using  Image 
Theory  to  emulate  the  decision  process  in  interactive  maintenance  computer  products  in  the  future. 

There  is  very  minimal  risk  that  I  might  experience  fatigue,  stress,  or  headaches  from 
using  the  interface.  This  risk  should  be  no  more  than  playing  a  video  game. 

Any  information  about  me  obtained  from  this  study  will  be  kept  strictly  confidential.  No 
names  or  other  such  personal  identifiers  will  be  used  on  the  surveys  or  linked  to  the  surveys. 
Subject  identification  numbers  will  be  used  to  link  performance  data  to  survey  data.  Performance 
data  will  be  securely  located  on  the  principal  investigator’s  personal  computer,  which  is  password 
protected.  I  will  not  be  identified  in  any  report  or  publication. 

I  acknowledge  that  this  study  has  been  explained  to  me,  and  that  the  principal 
investigator  has  discussed  the  possible  risks.  I  certify  that  I  have  been  given  the  opportunity  to 
have  all  of  my  questions  regarding  this  study  answered. 

I  am  free  to  refuse  to  participate  in  this  study,  or  to  withdraw  at  any  time.  My  decision  to 
participate  or  to  not  participate  will  not  adversely  affect  me  in  any  way. 

My  signature  below  means  that  I  have  freely  agreed  to  participate  in  this  investigational 

study. 
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APPENDIX  B 


Post  Experimental  Subjective  Questionnaire 


Subject  Number: _  Date: 


1.  What  is  your  gender?  M  F 

2,  I  have  had _ years  of  experience  in  the  maintenance  career  field. 


3. 

Do  you  have  visual  impairment  such  that  you  would  not  be  able  to  view  a  standard  15-inch  color 

4. 

monitor  to  perform  a  computer-based  task? 

Do  you  have  experience: 

Y 

N 

4A. 

Choosing  aircraft  for  deployment  or  special  duty? 

Y 

N 

4B. 

Scheduling  aircraft  for  repair  and  inspection? 

Y 

N 

4C. 

Determining  priority  of  repair  actions? 

Y 

N 

4D. 

Serving  as  part  of  a  deployed  maintenance  unit? 

Y 

N 

4E. 

Using  a  computer  to  process  information? 

Y 

N 

4F. 

Using  a  CAMS-like  computer  program  to  keep  track  of  aircraft  data? 

Y 

N 

4G. 

Viewing  and  assessing  the  impact  of  aircraft  repair  histories? 

Y 

N 

4H. 

Determining  if  an  aircraft  is  combat  ready? 

Y 

N 

5. 

Did  the  Information-Presentation-Only  (IPO)  computer  program  adequately  reflect  the  following: 

5A. 

Aircraft  Status 

Y 

N 

5B. 

Aircraft  Repair  History 

Y 

N 

5C. 

Scheduled  Maintenance  Items 

Y 

N 

5D. 

Unscheduled  Maintenance  Items 

Y 

N 

5E. 

Aircraft  Location 

Y 

N 

5F. 

Aircraft  Configuration 

Y 

N 

5G. 

Phase  Inspection  Information 

Y 

N 

5H.  If  you  circled  ‘N’  in  any  of  the  above  section  4  items,  please  explain  what  was  lacking  in  the  space 
provided. 


6.  For  the  IPO  tool  only,  was  there  information  that  was  necessary  to  choose  aircraft  for  deployment  that 
was  not  provided  within  the  context  of  the  mission  scenario?  Y  N 

If  you  selected  ‘Y\  please  explain  the  needed  items  in  the  space  provided? 
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7.  For  the  IPO:  rate  the  confidence  you  have  that  the  solution  you  generated  is  correct 

(Very  Low  Confidence)- 1  2  3  4  5  6  7-(Very  High  Confidence) 

8.  Did  the  expert  knowledge-based  Interactive  Decision  Support  System  (IDSS)  provide  better 
understanding  than  the  IPO  when  compiling  the  recommended  aircraft  list? 

Y  N 

9.  Did  the  expert  knowledge-based  IDSS  provide  all  the  necessary  information  to  perform  the  task? 

Y  N 

If  not,  provide  examples  in  the  space  below: 


10.  For  the  expert  knowledge-based  IDSS:  were  computer  suggestions  helpful  in  performing  the  task? 

Y  N 

11.  For  the  expert  knowledge-based  IDSS:  rate  the  confidence  you  have  that  the  solution  you  generated  is 

correct  (Very  Low  Confidence)- 1  2  3  4  5  6  7-(Very  High  Confidence) 

12.  Do  you  prefer  to  generate  the  aircraft  list  yourself  or  have  the  computer  help  you  with  suggestions? 

Generate  the  list  myself  I  like  the  computer  suggestions  Doesn’t  matter  to  me 

13.  Did  the  guidance  provided  by  the  expert  knowledge-based  IDSS  help  you  or  hinder  the  process? 

Help  Hinder 

14.  Which  tool  would  you  prefer  for  selecting  aircraft  for  deployment?  IDSS  IPO  Neither 

15.  Please  provide  CONSTRUCTIVE  comments  to  the  experiment  staff  on  how  to  improve  the  tools  in  the 
future. 
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Definitely  the  IPO  Mostly  IPO  Marginally  Neutral  Marginally  IDSS  Mostly  Definitely  the 

IPO  Response  IDSS  IDSS 

- 1 -  - 2 -  - 3 -  - 4  - . . . 5 - - - 6 -  - 7 - 


Use  the  scale  above,  rate  the  test  conditions  based  on  the  following: 


16.  Produced  a  solution  that  I  feel  comfortable  using  in  the  field. 

17.  Allowed  me  to  make  a  better  decision. 

18.  Allowed  me  to  fully  comprehend  the  aircraft  information. 

19.  Increased  my  understanding  of  the  aircraft  fleet  status. 

20.  Increased  my  understanding  of  my  own  decision  process. 

21.  Increased  my  trust  in  the  final  decision. 

22.  Provided  more  freedom  to  explore  the  information  thoroughly. 

23.  Inhibited  my  ability  to  understand  the  aircraft  fleet  status. 

24.  Inhibited  my  understanding  of  my  own  decision  process. 

25.  Produced  a  solution  that  I  would  not  use. 

26.  Prevented  me  from  full  comprehension  of  the  aircraft  information 

27.  Inhibited  my  freedom  to  explore  the  information  thoroughly 

28.  Prevented  me  from  making  a  good  decision. 

29.  Disrupted  the  process  I  prefer  to  use. 


1 — 2 — 3 — 4 — 5 — 6 — 7 
1 — 2 — 3 — 4 — 5 — 6 — 7 
1 — 2 — 3 — 4 — 5 — 6 — 7 
1 — 2 — 3 — 4 — 5 — 6 — 7 
1 — 2 — 3 — 4 — 5—6 — 7 
1 — 2 — 3 — 4 — 5 — 6 — 7 
1  — 2 — 3 — 4 — 5 — 6 — 7 
1  — 2 — 3  — 4 — 5 — 6 — 7 
1—2— 3— 4— 5— 6— 7 
1 — 2 — 3 — 4 — 5 — 6 — 7 
1 — 2 — 3 — 4—5 — 6 — 7 
1  — 2 — 3 — 4 — 5 — 6 — 7 
1  — 2 — 3 — 4 — 5 — 6 — 7 
1 — 2 — 3 — 4 — 5 — 6 — 7 
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APPENDIX  C 


WRITTEN  COMMENTS 


Subject 

Comments 

1. 

Tasking  to  choose  deployment  aircraft  and  schedule  aircraft  for 

repair  and  inspection  was  done  at  a  higher  level 

2. 

The  small  red  blocks  [in  the  scheduled  inspection  frame]  didn’t  tell 

what  the  aircraft  is  going  to  be  down  for 

3. 

Tail  numbers  overlap  and  cannot  be  read 

Could  not  find  aircraft  317  on  last  page 

Would  like  to  see  the  information  displayed  by  aircraft  tail  number, 

as  an  option. 

The  reason  for  my  low  confidence  was  a  programming  difficulty.  It 

would  not  select  and  move  tail  numbers  properly 

5. 

Need  WUCs  [Work  Unit  Codes] 

Need  to  have  in  the  scenario  the  number  of  weeks  deployed  so 

you  could  compare  like  information  on  the  schedule 

6. 

Make  program  more  user  friendly,  i.e.  click  and  drag,  etc. 

7. 

1  preferred  the  IPO  only  because  it  was  confusing  to  look  back  and 

forth  between  two  sets  of  number  lists  to  make  comparisons 

8. 

A  very  good  training  and  time  to  train  in  use  and  implementation. 

We  at  the  Maint.  Level  get  the  one  class  and  do  it  syndrome 

without  allowing  for  time  to  build  confidence  for  use. 

10. 

Unable  to  see  scheduled  maintenance  tasks.  On  repair  history 

chart  could  not  scroll  down. 

Need  to  display  upcoming  inspection  requirements 
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11.  On  aircraft  status,  I  would  have  an  ETIC  associated  with 
PMC/NMC 

On  history,  it  was  confusing  as  to  what  discrepancy  currently 
exists 

On  Sch.  Maint.,  put  how  long  until  the  work  is  complete 
Window  displaying  phase  time  is  too  small  and  required  me  to 
scroll  back  and  forth  to  find  the  data. 

12.  On  Sch  Maint.,  items  were  only  displayed  as  bars  on  a  chart; 
would  need  to  know  what  the  bars  represented 

Need  better  explanation  of  corrective  actions  on  USM 

Need  to  know  specific  equipment  installed  on  aircraft 

Need  to  know  if  ferry  time  is  to  be  included  or  excluded  from  100 

hour  requirement. 

What  will  the  projected  UTE  rate  be  at  the  deployment  location 
What  will  the  average  sortie  duration  (ASD)  be  while  deployed 
Not  enough  detail  on  the  scheduled  maintenance  chart 
Some  information  is  confusing 

Operations  variables  such  as  average  sortie  duration,  mission 
variation  requirements  and  etc.  Should  also  be  available. 

I  like  the  idea  of  a  Wizard  or  help  mate  to  make  decisions.  It 
would  be  better  if  all  the  information  was  available  on  one  screen 
and  I  could  pick  and  choose  which  information  to  inflate  for  closer 
inspection.  The  computer  suggestions  were  very  helpful  and 
should  be  provided  when  asked.  I  favor  a  combination  of  the  two 
different  conditions  so  that  I  can  freely  explore  the  information 
while  also  getting  some  computer  help. 
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APPENDIX  D 


Expert  Ranking  of  Subject  Responses 
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